Reclaiming the Loop: From the Consensus Trap to Pluralistic Data Annotation

Sheza Munir
Proceedings of the The 39th Canadian Conference on Artificial Intelligence, PMLR 318:1216-1218, 2026.

Abstract

This research challenges the dominant “ground truth” paradigm in machine learning, arguing that current annotation practices suppress meaningful human disagreement in favor of artificial consensus. It identifies two structural failures in annotation pipelines: the allocation gap (mismatch between annotator identity and data context) and the representation gap (erasure of nuance during label aggregation). The proposed solution introduces a pluralistic annotation infrastructure that incorporates identity-aware task assignment and rationale-aware aggregation to preserve lived experience and dissent. By reframing disagreement as a high-fidelity epistemic signal rather than noise, the work advances a model of situated knowledge stewardship aimed at promoting epistemic justice in AI systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v318-munir26b, title = {Reclaiming the Loop: From the Consensus Trap to Pluralistic Data Annotation}, author = {Munir, Sheza}, booktitle = {Proceedings of the The 39th Canadian Conference on Artificial Intelligence}, pages = {1216--1218}, year = {2026}, editor = {Bouzar-Benlabiod, Lydia and Leung, Carson}, volume = {318}, series = {Proceedings of Machine Learning Research}, month = {25--29 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v318/main/assets/munir26b/munir26b.pdf}, url = {https://proceedings.mlr.press/v318/munir26b.html}, abstract = {This research challenges the dominant “ground truth” paradigm in machine learning, arguing that current annotation practices suppress meaningful human disagreement in favor of artificial consensus. It identifies two structural failures in annotation pipelines: the allocation gap (mismatch between annotator identity and data context) and the representation gap (erasure of nuance during label aggregation). The proposed solution introduces a pluralistic annotation infrastructure that incorporates identity-aware task assignment and rationale-aware aggregation to preserve lived experience and dissent. By reframing disagreement as a high-fidelity epistemic signal rather than noise, the work advances a model of situated knowledge stewardship aimed at promoting epistemic justice in AI systems.} }
Endnote
%0 Conference Paper %T Reclaiming the Loop: From the Consensus Trap to Pluralistic Data Annotation %A Sheza Munir %B Proceedings of the The 39th Canadian Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Lydia Bouzar-Benlabiod %E Carson Leung %F pmlr-v318-munir26b %I PMLR %P 1216--1218 %U https://proceedings.mlr.press/v318/munir26b.html %V 318 %X This research challenges the dominant “ground truth” paradigm in machine learning, arguing that current annotation practices suppress meaningful human disagreement in favor of artificial consensus. It identifies two structural failures in annotation pipelines: the allocation gap (mismatch between annotator identity and data context) and the representation gap (erasure of nuance during label aggregation). The proposed solution introduces a pluralistic annotation infrastructure that incorporates identity-aware task assignment and rationale-aware aggregation to preserve lived experience and dissent. By reframing disagreement as a high-fidelity epistemic signal rather than noise, the work advances a model of situated knowledge stewardship aimed at promoting epistemic justice in AI systems.
APA
Munir, S.. (2026). Reclaiming the Loop: From the Consensus Trap to Pluralistic Data Annotation. Proceedings of the The 39th Canadian Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 318:1216-1218 Available from https://proceedings.mlr.press/v318/munir26b.html.

Related Material