[edit]
Reducing Poisson error can offset classification error: a technique to meet clinical performance requirements
Proceedings of the 4th Machine Learning for Health Symposium, PMLR 259:233-247, 2025.
Abstract
Medical machine learning algorithms are typically evaluated based on their object-level accuracy vs. that of skilled clinicians, a challenging bar since trained clinicians are usually better classifiers than ML models. However, this metric does not fully capture the realities and requirements of the actual clinical task: it neglects the fact that humans, even with perfect object-level accuracy, are subject to non-trivial error from the Poisson statistics of rare events, because clinical protocols often specify a remarkably small sample size due to the exigencies of clinical work. For example, to quantitate malaria on a thin blood film a clinician examines only 2000 red blood cells (0.0004 uL), which can yield large Poisson variation in the actual number of parasites present, so that a perfect human’s count can differ substantially from the true average load. In contrast, an ML system may be less accurate on an object detection level, but it may also have the option to examine much more blood (e.g. 0.1 uL, or 250x). Thus, while its parasite identification error is higher, the Poisson variability of its estimate is lower due to larger sample size. For both ML systems and humans, clinical performance depends on a combination of these two types of error. To qualify for clinical deployment, an ML system’s performance must match current standard of care, typically a demanding target. To achieve this, it may be possible to offset a system’s imperfect accuracy by increasing its sample size to reduce Poisson error, and thus attain the same net clinical performance as a perfectly accurate human limited by protocols with smaller sample size. In this paper, we analyse the mathematics of the relationship between Poisson error, classification error, and total error. This mathematical approach enables teams (software and hardware) optimizing ML systems to leverage a relative strength (larger sample sizes) to offset a relative weakness (classification accuracy). We illustrate the methods with two concrete examples: diagnosis and quantitation of malaria on blood films.