A Multi-Agent System for Feedback Quality Assurance: Evaluate, Revise, and Validate at Scale

Okan Bulut, Cole Walsh
Proceedings of the Impactful and Responsible AI Systems for Education Workshop, PMLR 339:1-11, 2026.

Abstract

Large language models (LLMs) have shown considerable promise for automatic feedback generation (AFG) in educational settings, yet their inaccuracies and unpredictable failures underscore the need for robust quality assurance (QA) before deployment at scale. Traditional QA approaches relying on human raters are inherently time-consuming, costly, and difficult to scale. This study extends the LLM-as-judge paradigm beyond evaluation to introduce a multi-agent QA system that not only assesses but autonomously revises LLM-generated feedback until predefined quality standards are met. The system comprises seven specialized agents operating in a closed evaluate–revise–re-evaluate loop. Skill-level feedback from a formative assessment measuring professional competencies ($n = 841$) is evaluated against six quality criteria using chain-of-thought prompting, and revised using role-based prompting where quality thresholds are not met. The system achieved a mean QA total of 5.99 out of 6 across all records, approved 99.0% of feedback within three iterations, and escalated only 1.0% of cases for human review. A single-agent baseline is compared against the proposed system to assess the contribution of the multi-agent architecture. The baseline performed comparably on five of the six criteria but failed markedly on second-person language (32.1% vs. 100%) and escalated 68.0% of records, compared to 1.0% in the proposed system, demonstrating that distributing QA responsibilities across specialized agents substantially improves both evaluation precision and revision effectiveness. A criterion-level stress test on 180 distorted records demonstrated a 100% correction rate for all detected violations and a 0% false positive rate, with detection rates ranging from 80.0% for actionability to 100% for second-person language. Human expert review of flagged and revised cases largely confirmed the system’s judgments, though reviewers noted that certain linguistic corrections, particularly those involving punctuation, were unlikely to meaningfully affect student comprehension. These findings establish the viability of multi-agent LLM systems as scalable, automated QA and remediation pipelines for AFG.

Cite this Paper

BibTeX
@InProceedings{pmlr-v339-bulut26a, title = {A Multi-Agent System for Feedback Quality Assurance: Evaluate, Revise, and Validate at Scale}, author = {Bulut, Okan and Walsh, Cole}, booktitle = {Proceedings of the Impactful and Responsible AI Systems for Education Workshop}, pages = {1--11}, year = {2026}, editor = {Basu Mallick, Debshila and Woodhead, Simon and Wang, Zichao and Ananda, Muktha and Burstein, Jill and Murphy, April}, volume = {339}, series = {Proceedings of Machine Learning Research}, month = {28 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v339/main/assets/bulut26a/bulut26a.pdf}, url = {https://proceedings.mlr.press/v339/bulut26a.html}, abstract = {Large language models (LLMs) have shown considerable promise for automatic feedback generation (AFG) in educational settings, yet their inaccuracies and unpredictable failures underscore the need for robust quality assurance (QA) before deployment at scale. Traditional QA approaches relying on human raters are inherently time-consuming, costly, and difficult to scale. This study extends the LLM-as-judge paradigm beyond evaluation to introduce a multi-agent QA system that not only assesses but autonomously revises LLM-generated feedback until predefined quality standards are met. The system comprises seven specialized agents operating in a closed evaluate–revise–re-evaluate loop. Skill-level feedback from a formative assessment measuring professional competencies ($n = 841$) is evaluated against six quality criteria using chain-of-thought prompting, and revised using role-based prompting where quality thresholds are not met. The system achieved a mean QA total of 5.99 out of 6 across all records, approved 99.0% of feedback within three iterations, and escalated only 1.0% of cases for human review. A single-agent baseline is compared against the proposed system to assess the contribution of the multi-agent architecture. The baseline performed comparably on five of the six criteria but failed markedly on second-person language (32.1% vs. 100%) and escalated 68.0% of records, compared to 1.0% in the proposed system, demonstrating that distributing QA responsibilities across specialized agents substantially improves both evaluation precision and revision effectiveness. A criterion-level stress test on 180 distorted records demonstrated a 100% correction rate for all detected violations and a 0% false positive rate, with detection rates ranging from 80.0% for actionability to 100% for second-person language. Human expert review of flagged and revised cases largely confirmed the system’s judgments, though reviewers noted that certain linguistic corrections, particularly those involving punctuation, were unlikely to meaningfully affect student comprehension. These findings establish the viability of multi-agent LLM systems as scalable, automated QA and remediation pipelines for AFG.} }
Endnote
%0 Conference Paper %T A Multi-Agent System for Feedback Quality Assurance: Evaluate, Revise, and Validate at Scale %A Okan Bulut %A Cole Walsh %B Proceedings of the Impactful and Responsible AI Systems for Education Workshop %C Proceedings of Machine Learning Research %D 2026 %E Debshila Basu Mallick %E Simon Woodhead %E Zichao Wang %E Muktha Ananda %E Jill Burstein %E April Murphy %F pmlr-v339-bulut26a %I PMLR %P 1--11 %U https://proceedings.mlr.press/v339/bulut26a.html %V 339 %X Large language models (LLMs) have shown considerable promise for automatic feedback generation (AFG) in educational settings, yet their inaccuracies and unpredictable failures underscore the need for robust quality assurance (QA) before deployment at scale. Traditional QA approaches relying on human raters are inherently time-consuming, costly, and difficult to scale. This study extends the LLM-as-judge paradigm beyond evaluation to introduce a multi-agent QA system that not only assesses but autonomously revises LLM-generated feedback until predefined quality standards are met. The system comprises seven specialized agents operating in a closed evaluate–revise–re-evaluate loop. Skill-level feedback from a formative assessment measuring professional competencies ($n = 841$) is evaluated against six quality criteria using chain-of-thought prompting, and revised using role-based prompting where quality thresholds are not met. The system achieved a mean QA total of 5.99 out of 6 across all records, approved 99.0% of feedback within three iterations, and escalated only 1.0% of cases for human review. A single-agent baseline is compared against the proposed system to assess the contribution of the multi-agent architecture. The baseline performed comparably on five of the six criteria but failed markedly on second-person language (32.1% vs. 100%) and escalated 68.0% of records, compared to 1.0% in the proposed system, demonstrating that distributing QA responsibilities across specialized agents substantially improves both evaluation precision and revision effectiveness. A criterion-level stress test on 180 distorted records demonstrated a 100% correction rate for all detected violations and a 0% false positive rate, with detection rates ranging from 80.0% for actionability to 100% for second-person language. Human expert review of flagged and revised cases largely confirmed the system’s judgments, though reviewers noted that certain linguistic corrections, particularly those involving punctuation, were unlikely to meaningfully affect student comprehension. These findings establish the viability of multi-agent LLM systems as scalable, automated QA and remediation pipelines for AFG.
APA
Bulut, O. & Walsh, C.. (2026). A Multi-Agent System for Feedback Quality Assurance: Evaluate, Revise, and Validate at Scale. Proceedings of the Impactful and Responsible AI Systems for Education Workshop, in Proceedings of Machine Learning Research 339:1-11 Available from https://proceedings.mlr.press/v339/bulut26a.html.

Related Material