Position: Mechanisms for Aggregated Individual Reporting Should be Established for Post-Deployment Evaluation

Jessica Dai, Inioluwa Deborah Raji, Benjamin Recht, Irene Y. Chen
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:169923-169938, 2026.

Abstract

The need for developing model evaluations beyond static benchmarking, especially in the post-deployment phase, is now well-understood. At the same time, concerns about the concentration of power in deployed AI systems have sparked a keen interest in "democratic" or "public" AI. In this work, we bring these two ideas together by proposing mechanisms for aggregated individual reporting (AIR), a framework for post-deployment evaluation that relies on individual reports from the public. An AIR mechanism allows those who interact with a specific, deployed (AI) system to report when they feel that they may have experienced something problematic; these reports are then aggregated over time, with the goal of evaluating the relevant system in a fine-grained manner. This position paper argues that individual experiences should be understood as an integral part of post-deployment evaluation, and that the scope of our proposed aggregated individual reporting mechanism is a practical path to that end. On the one hand, individual reporting can identify substantively novel insights about safety and performance; on the other, aggregation can be uniquely useful for informing action. From a normative perspective, the post-deployment phase completes a missing piece in the conversation about "democratic" AI. As a pathway to implementation, we provide a workflow of concrete design decisions and pointers to areas for future research.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-dai26m, title = {Position: Mechanisms for Aggregated Individual Reporting Should be Established for Post-Deployment Evaluation}, author = {Dai, Jessica and Raji, Inioluwa Deborah and Recht, Benjamin and Chen, Irene Y.}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {169923--169938}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/dai26m/dai26m.pdf}, url = {https://proceedings.mlr.press/v306/dai26m.html}, abstract = {The need for developing model evaluations beyond static benchmarking, especially in the post-deployment phase, is now well-understood. At the same time, concerns about the concentration of power in deployed AI systems have sparked a keen interest in "democratic" or "public" AI. In this work, we bring these two ideas together by proposing mechanisms for aggregated individual reporting (AIR), a framework for post-deployment evaluation that relies on individual reports from the public. An AIR mechanism allows those who interact with a specific, deployed (AI) system to report when they feel that they may have experienced something problematic; these reports are then aggregated over time, with the goal of evaluating the relevant system in a fine-grained manner. This position paper argues that individual experiences should be understood as an integral part of post-deployment evaluation, and that the scope of our proposed aggregated individual reporting mechanism is a practical path to that end. On the one hand, individual reporting can identify substantively novel insights about safety and performance; on the other, aggregation can be uniquely useful for informing action. From a normative perspective, the post-deployment phase completes a missing piece in the conversation about "democratic" AI. As a pathway to implementation, we provide a workflow of concrete design decisions and pointers to areas for future research.} }
Endnote
%0 Conference Paper %T Position: Mechanisms for Aggregated Individual Reporting Should be Established for Post-Deployment Evaluation %A Jessica Dai %A Inioluwa Deborah Raji %A Benjamin Recht %A Irene Y. Chen %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-dai26m %I PMLR %P 169923--169938 %U https://proceedings.mlr.press/v306/dai26m.html %V 306 %X The need for developing model evaluations beyond static benchmarking, especially in the post-deployment phase, is now well-understood. At the same time, concerns about the concentration of power in deployed AI systems have sparked a keen interest in "democratic" or "public" AI. In this work, we bring these two ideas together by proposing mechanisms for aggregated individual reporting (AIR), a framework for post-deployment evaluation that relies on individual reports from the public. An AIR mechanism allows those who interact with a specific, deployed (AI) system to report when they feel that they may have experienced something problematic; these reports are then aggregated over time, with the goal of evaluating the relevant system in a fine-grained manner. This position paper argues that individual experiences should be understood as an integral part of post-deployment evaluation, and that the scope of our proposed aggregated individual reporting mechanism is a practical path to that end. On the one hand, individual reporting can identify substantively novel insights about safety and performance; on the other, aggregation can be uniquely useful for informing action. From a normative perspective, the post-deployment phase completes a missing piece in the conversation about "democratic" AI. As a pathway to implementation, we provide a workflow of concrete design decisions and pointers to areas for future research.
APA
Dai, J., Raji, I.D., Recht, B. & Chen, I.Y.. (2026). Position: Mechanisms for Aggregated Individual Reporting Should be Established for Post-Deployment Evaluation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:169923-169938 Available from https://proceedings.mlr.press/v306/dai26m.html.

Related Material