SySDEM - Synthetic and Stratified Degradations for Evaluating Metrics for Long-Form Text in Medical Domain

Naveen Jafer Nizar, Qinlan Shen, Sumana Srivatsa, Krishnaram Kenthapadi
Proceedings of the Fifth Machine Learning for Health Symposium, PMLR 297:1075-1095, 2026.

Abstract

The evaluation of long-form text in the medical domain is increasingly reliant on automated metrics. However, the reliability of these metrics themselves is often assumed rather than rigorously tested, especially when long-form generations are the expected output. We address this gap by proposing {SySDEM} - Synthetic and Stratified Degradations for Evaluating Metrics, a framework to evaluate the quality of reference-based evaluation metrics. Using this framework, we demonstrate a method that iteratively perturbs candidate texts to assess the sensitivity and discrimination power of reference-based text evaluation metrics. Through experiments on the {ACI}-Bench clinical note generation dataset, we demonstrate the importance of evaluating evaluation metrics for long-form text, highlighting the need for robust validation methodologies.

Cite this Paper

BibTeX
@InProceedings{pmlr-v297-nizar26a, title = {{SySDEM} - Synthetic and Stratified Degradations for Evaluating Metrics for Long-Form Text in Medical Domain}, author = {Nizar, Naveen Jafer and Shen, Qinlan and Srivatsa, Sumana and Kenthapadi, Krishnaram}, booktitle = {Proceedings of the Fifth Machine Learning for Health Symposium}, pages = {1075--1095}, year = {2026}, editor = {Argaw, Peniel and Zhang, Haoran and Jabbour, Sarah and Chandak, Payal and Ji, Jerry and Mukherjee, Sumit and Salaudeen, Olawale and Chang, Trenton and Healey, Elizabeth and Gröger, Fabian and Adibi, Amin and Hegselmann, Stefan and Wild, Benjamin and Noori, Ayush}, volume = {297}, series = {Proceedings of Machine Learning Research}, month = {13--14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v297/main/assets/nizar26a/nizar26a.pdf}, url = {https://proceedings.mlr.press/v297/nizar26a.html}, abstract = {The evaluation of long-form text in the medical domain is increasingly reliant on automated metrics. However, the reliability of these metrics themselves is often assumed rather than rigorously tested, especially when long-form generations are the expected output. We address this gap by proposing {SySDEM} - Synthetic and Stratified Degradations for Evaluating Metrics, a framework to evaluate the quality of reference-based evaluation metrics. Using this framework, we demonstrate a method that iteratively perturbs candidate texts to assess the sensitivity and discrimination power of reference-based text evaluation metrics. Through experiments on the {ACI}-Bench clinical note generation dataset, we demonstrate the importance of evaluating evaluation metrics for long-form text, highlighting the need for robust validation methodologies.} }
Endnote
%0 Conference Paper %T SySDEM - Synthetic and Stratified Degradations for Evaluating Metrics for Long-Form Text in Medical Domain %A Naveen Jafer Nizar %A Qinlan Shen %A Sumana Srivatsa %A Krishnaram Kenthapadi %B Proceedings of the Fifth Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2026 %E Peniel Argaw %E Haoran Zhang %E Sarah Jabbour %E Payal Chandak %E Jerry Ji %E Sumit Mukherjee %E Olawale Salaudeen %E Trenton Chang %E Elizabeth Healey %E Fabian Gröger %E Amin Adibi %E Stefan Hegselmann %E Benjamin Wild %E Ayush Noori %F pmlr-v297-nizar26a %I PMLR %P 1075--1095 %U https://proceedings.mlr.press/v297/nizar26a.html %V 297 %X The evaluation of long-form text in the medical domain is increasingly reliant on automated metrics. However, the reliability of these metrics themselves is often assumed rather than rigorously tested, especially when long-form generations are the expected output. We address this gap by proposing {SySDEM} - Synthetic and Stratified Degradations for Evaluating Metrics, a framework to evaluate the quality of reference-based evaluation metrics. Using this framework, we demonstrate a method that iteratively perturbs candidate texts to assess the sensitivity and discrimination power of reference-based text evaluation metrics. Through experiments on the {ACI}-Bench clinical note generation dataset, we demonstrate the importance of evaluating evaluation metrics for long-form text, highlighting the need for robust validation methodologies.
APA
Nizar, N.J., Shen, Q., Srivatsa, S. & Kenthapadi, K.. (2026). SySDEM - Synthetic and Stratified Degradations for Evaluating Metrics for Long-Form Text in Medical Domain. Proceedings of the Fifth Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 297:1075-1095 Available from https://proceedings.mlr.press/v297/nizar26a.html.

Related Material