Reducing Poisson error can offset classification error: a technique to meet clinical performance requirements

Charles B. Delahunt, Courosh Mehanian, Matthew P. Horning
Proceedings of the 4th Machine Learning for Health Symposium, PMLR 259:233-247, 2025.

Abstract

Medical machine learning algorithms are typically evaluated based on their object-level accuracy vs. that of skilled clinicians, a challenging bar since trained clinicians are usually better classifiers than ML models. However, this metric does not fully capture the realities and requirements of the actual clinical task: it neglects the fact that humans, even with perfect object-level accuracy, are subject to non-trivial error from the Poisson statistics of rare events, because clinical protocols often specify a remarkably small sample size due to the exigencies of clinical work. For example, to quantitate malaria on a thin blood film a clinician examines only 2000 red blood cells (0.0004 uL), which can yield large Poisson variation in the actual number of parasites present, so that a perfect human’s count can differ substantially from the true average load. In contrast, an ML system may be less accurate on an object detection level, but it may also have the option to examine much more blood (e.g. 0.1 uL, or 250x). Thus, while its parasite identification error is higher, the Poisson variability of its estimate is lower due to larger sample size. For both ML systems and humans, clinical performance depends on a combination of these two types of error. To qualify for clinical deployment, an ML system’s performance must match current standard of care, typically a demanding target. To achieve this, it may be possible to offset a system’s imperfect accuracy by increasing its sample size to reduce Poisson error, and thus attain the same net clinical performance as a perfectly accurate human limited by protocols with smaller sample size. In this paper, we analyse the mathematics of the relationship between Poisson error, classification error, and total error. This mathematical approach enables teams (software and hardware) optimizing ML systems to leverage a relative strength (larger sample sizes) to offset a relative weakness (classification accuracy). We illustrate the methods with two concrete examples: diagnosis and quantitation of malaria on blood films.

Cite this Paper

BibTeX
@InProceedings{pmlr-v259-delahunt25a, title = {Reducing Poisson error can offset classification error: a technique to meet clinical performance requirements}, author = {Delahunt, Charles B. and Mehanian, Courosh and Horning, Matthew P.}, booktitle = {Proceedings of the 4th Machine Learning for Health Symposium}, pages = {233--247}, year = {2025}, editor = {Hegselmann, Stefan and Zhou, Helen and Healey, Elizabeth and Chang, Trenton and Ellington, Caleb and Mhasawade, Vishwali and Tonekaboni, Sana and Argaw, Peniel and Zhang, Haoran}, volume = {259}, series = {Proceedings of Machine Learning Research}, month = {15--16 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v259/main/assets/delahunt25a/delahunt25a.pdf}, url = {https://proceedings.mlr.press/v259/delahunt25a.html}, abstract = {Medical machine learning algorithms are typically evaluated based on their object-level accuracy vs. that of skilled clinicians, a challenging bar since trained clinicians are usually better classifiers than ML models. However, this metric does not fully capture the realities and requirements of the actual clinical task: it neglects the fact that humans, even with perfect object-level accuracy, are subject to non-trivial error from the Poisson statistics of rare events, because clinical protocols often specify a remarkably small sample size due to the exigencies of clinical work. For example, to quantitate malaria on a thin blood film a clinician examines only 2000 red blood cells (0.0004 uL), which can yield large Poisson variation in the actual number of parasites present, so that a perfect human’s count can differ substantially from the true average load. In contrast, an ML system may be less accurate on an object detection level, but it may also have the option to examine much more blood (e.g. 0.1 uL, or 250x). Thus, while its parasite identification error is higher, the Poisson variability of its estimate is lower due to larger sample size. For both ML systems and humans, clinical performance depends on a combination of these two types of error. To qualify for clinical deployment, an ML system’s performance must match current standard of care, typically a demanding target. To achieve this, it may be possible to offset a system’s imperfect accuracy by increasing its sample size to reduce Poisson error, and thus attain the same net clinical performance as a perfectly accurate human limited by protocols with smaller sample size. In this paper, we analyse the mathematics of the relationship between Poisson error, classification error, and total error. This mathematical approach enables teams (software and hardware) optimizing ML systems to leverage a relative strength (larger sample sizes) to offset a relative weakness (classification accuracy). We illustrate the methods with two concrete examples: diagnosis and quantitation of malaria on blood films.} }
Endnote
%0 Conference Paper %T Reducing Poisson error can offset classification error: a technique to meet clinical performance requirements %A Charles B. Delahunt %A Courosh Mehanian %A Matthew P. Horning %B Proceedings of the 4th Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2025 %E Stefan Hegselmann %E Helen Zhou %E Elizabeth Healey %E Trenton Chang %E Caleb Ellington %E Vishwali Mhasawade %E Sana Tonekaboni %E Peniel Argaw %E Haoran Zhang %F pmlr-v259-delahunt25a %I PMLR %P 233--247 %U https://proceedings.mlr.press/v259/delahunt25a.html %V 259 %X Medical machine learning algorithms are typically evaluated based on their object-level accuracy vs. that of skilled clinicians, a challenging bar since trained clinicians are usually better classifiers than ML models. However, this metric does not fully capture the realities and requirements of the actual clinical task: it neglects the fact that humans, even with perfect object-level accuracy, are subject to non-trivial error from the Poisson statistics of rare events, because clinical protocols often specify a remarkably small sample size due to the exigencies of clinical work. For example, to quantitate malaria on a thin blood film a clinician examines only 2000 red blood cells (0.0004 uL), which can yield large Poisson variation in the actual number of parasites present, so that a perfect human’s count can differ substantially from the true average load. In contrast, an ML system may be less accurate on an object detection level, but it may also have the option to examine much more blood (e.g. 0.1 uL, or 250x). Thus, while its parasite identification error is higher, the Poisson variability of its estimate is lower due to larger sample size. For both ML systems and humans, clinical performance depends on a combination of these two types of error. To qualify for clinical deployment, an ML system’s performance must match current standard of care, typically a demanding target. To achieve this, it may be possible to offset a system’s imperfect accuracy by increasing its sample size to reduce Poisson error, and thus attain the same net clinical performance as a perfectly accurate human limited by protocols with smaller sample size. In this paper, we analyse the mathematics of the relationship between Poisson error, classification error, and total error. This mathematical approach enables teams (software and hardware) optimizing ML systems to leverage a relative strength (larger sample sizes) to offset a relative weakness (classification accuracy). We illustrate the methods with two concrete examples: diagnosis and quantitation of malaria on blood films.
APA
Delahunt, C.B., Mehanian, C. & Horning, M.P.. (2025). Reducing Poisson error can offset classification error: a technique to meet clinical performance requirements. Proceedings of the 4th Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 259:233-247 Available from https://proceedings.mlr.press/v259/delahunt25a.html.

Related Material