Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding

Lina Zhang, Tonmoy Monsoor, Peizheng Li, Jiarui Cui, Xinyi Peng, Chong Han, Prateik Sinha, Siyuan Dai, Jessica Nichole Pasqua, Colin M Mccrimmon, Weiting Liu, Hailey Marie Miranda, Bing Hu, Xiangting Wu, Tengyou Xu, Chunhan Li, Jiaye Tian, Jiarui Tang, Detao Ma, Lingye Kong, Junnan Lyu, Jungang Li, Yan Zan, Junhua Huang, Rajarshi Mazumder, Vwani Roychowdhury
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:157800-157840, 2026.

Abstract

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in general video understanding, their capacity to interpret involuntary, and spatio-temporally evolving pathologic motor behaviors such as seizure semiology remains largely untested. To address this gap, we introduce Seizure-Semiology-Suite (S$^3$), a clinically grounded dataset and benchmark for fine-grained, structured seizure semiology understanding. The dataset includes 438 seizure videos annotated with over 35,000 dense labels covering 20 ILAE-defined semiological features. Building on this dataset, we propose a seven-task hierarchical benchmark that systematically evaluates MLLMs from low-level visual perception to temporal sequencing, narrative report generation, and seizure diagnosis. To enable clinically meaningful evaluation of generated reports, we further introduce the Report Quality Index for Seizure Semiology (Seizure-RQI). Extensive baselines across 11 open-weight MLLMs reveal systematic weaknesses in laterality reasoning, temporal localization, symptom sequencing, and clinically faithful reporting. We show that seizure-specific fine-tuning substantially improves performance across tasks, and that a two-stage neuro-symbolic framework achieves an F1 score of 0.96 on epileptic versus non-epileptic seizure classification. Seizure-Semiology-Suite establishes a rigorous benchmark for evaluating multimodal models in safety-critical medical video understanding and guides the development of clinically reliable, domain-adaptive multimodal intelligence. Our code is publicly available at https://github.com/LinaZhangUCLA/SeizureSemiologySuite.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-zhang26el, title = {Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding}, author = {Zhang, Lina and Monsoor, Tonmoy and Li, Peizheng and Cui, Jiarui and Peng, Xinyi and Han, Chong and Sinha, Prateik and Dai, Siyuan and Pasqua, Jessica Nichole and Mccrimmon, Colin M and Liu, Weiting and Miranda, Hailey Marie and Hu, Bing and Wu, Xiangting and Xu, Tengyou and Li, Chunhan and Tian, Jiaye and Tang, Jiarui and Ma, Detao and Kong, Lingye and Lyu, Junnan and Li, Jungang and Zan, Yan and Huang, Junhua and Mazumder, Rajarshi and Roychowdhury, Vwani}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {157800--157840}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhang26el/zhang26el.pdf}, url = {https://proceedings.mlr.press/v306/zhang26el.html}, abstract = {While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in general video understanding, their capacity to interpret involuntary, and spatio-temporally evolving pathologic motor behaviors such as seizure semiology remains largely untested. To address this gap, we introduce Seizure-Semiology-Suite (S$^3$), a clinically grounded dataset and benchmark for fine-grained, structured seizure semiology understanding. The dataset includes 438 seizure videos annotated with over 35,000 dense labels covering 20 ILAE-defined semiological features. Building on this dataset, we propose a seven-task hierarchical benchmark that systematically evaluates MLLMs from low-level visual perception to temporal sequencing, narrative report generation, and seizure diagnosis. To enable clinically meaningful evaluation of generated reports, we further introduce the Report Quality Index for Seizure Semiology (Seizure-RQI). Extensive baselines across 11 open-weight MLLMs reveal systematic weaknesses in laterality reasoning, temporal localization, symptom sequencing, and clinically faithful reporting. We show that seizure-specific fine-tuning substantially improves performance across tasks, and that a two-stage neuro-symbolic framework achieves an F1 score of 0.96 on epileptic versus non-epileptic seizure classification. Seizure-Semiology-Suite establishes a rigorous benchmark for evaluating multimodal models in safety-critical medical video understanding and guides the development of clinically reliable, domain-adaptive multimodal intelligence. Our code is publicly available at https://github.com/LinaZhangUCLA/SeizureSemiologySuite.} }
Endnote
%0 Conference Paper %T Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding %A Lina Zhang %A Tonmoy Monsoor %A Peizheng Li %A Jiarui Cui %A Xinyi Peng %A Chong Han %A Prateik Sinha %A Siyuan Dai %A Jessica Nichole Pasqua %A Colin M Mccrimmon %A Weiting Liu %A Hailey Marie Miranda %A Bing Hu %A Xiangting Wu %A Tengyou Xu %A Chunhan Li %A Jiaye Tian %A Jiarui Tang %A Detao Ma %A Lingye Kong %A Junnan Lyu %A Jungang Li %A Yan Zan %A Junhua Huang %A Rajarshi Mazumder %A Vwani Roychowdhury %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhang26el %I PMLR %P 157800--157840 %U https://proceedings.mlr.press/v306/zhang26el.html %V 306 %X While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in general video understanding, their capacity to interpret involuntary, and spatio-temporally evolving pathologic motor behaviors such as seizure semiology remains largely untested. To address this gap, we introduce Seizure-Semiology-Suite (S$^3$), a clinically grounded dataset and benchmark for fine-grained, structured seizure semiology understanding. The dataset includes 438 seizure videos annotated with over 35,000 dense labels covering 20 ILAE-defined semiological features. Building on this dataset, we propose a seven-task hierarchical benchmark that systematically evaluates MLLMs from low-level visual perception to temporal sequencing, narrative report generation, and seizure diagnosis. To enable clinically meaningful evaluation of generated reports, we further introduce the Report Quality Index for Seizure Semiology (Seizure-RQI). Extensive baselines across 11 open-weight MLLMs reveal systematic weaknesses in laterality reasoning, temporal localization, symptom sequencing, and clinically faithful reporting. We show that seizure-specific fine-tuning substantially improves performance across tasks, and that a two-stage neuro-symbolic framework achieves an F1 score of 0.96 on epileptic versus non-epileptic seizure classification. Seizure-Semiology-Suite establishes a rigorous benchmark for evaluating multimodal models in safety-critical medical video understanding and guides the development of clinically reliable, domain-adaptive multimodal intelligence. Our code is publicly available at https://github.com/LinaZhangUCLA/SeizureSemiologySuite.
APA
Zhang, L., Monsoor, T., Li, P., Cui, J., Peng, X., Han, C., Sinha, P., Dai, S., Pasqua, J.N., Mccrimmon, C.M., Liu, W., Miranda, H.M., Hu, B., Wu, X., Xu, T., Li, C., Tian, J., Tang, J., Ma, D., Kong, L., Lyu, J., Li, J., Zan, Y., Huang, J., Mazumder, R. & Roychowdhury, V.. (2026). Seizure-Semiology-Suite($S^3$): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:157800-157840 Available from https://proceedings.mlr.press/v306/zhang26el.html.

Related Material