Automatic Classifiers as Scientific Instruments: One Step Further Away from GroundTruth
[edit]
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:67576765, 2019.
Abstract
Automatic machine learningbased detectors of various psychological and social phenomena (e.g., emotion, stress, engagement) have great potential to advance basic science. However, when a detector d is trained to approximate an existing measurement tool (e.g., a questionnaire, observation protocol), then care must be taken when interpreting measurements collected using d since they are one step further removed from the under lying construct. We examine how the accuracy of d, as quantified by the correlation q of d’s out puts with the groundtruth construct U, impacts the estimated correlation between U (e.g., stress) and some other phenomenon V (e.g., academic performance). In particular: (1) We show that if the true correlation between U and V is r, then the expected sample correlation, over all vectors T n whose correlation with U is q, is qr. (2) We derive a formula for the probability that the sample correlation (over n subjects) using d is positive given that the true correlation is negative (and viceversa); this probability can be substantial (around 20  30%) for values of n and q that have been used in recent affective computing studies. (3) With the goal to reduce the variance of correlations estimated by an automatic detector, we show that training multiple neural networks d(1) , . . . , d(m) using different training architectures and hyperparameters for the same detection task provides only limited “coverage” of T^n.
Related Material


