Beyond the Black Box: Claim-Level Interpretability for Accountable Retrieval-Augmented Generation

Farjana Yesmin, Nusrat Shirmin
Proceedings of Fifth European Conference on Algorithmic Fairness, PMLR 350:165-177, 2026.

Abstract

Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains such as healthcare, legal aid, and public information services contexts where opacity in AI decision-support raises fundamental accountability and fairness concerns. Standard RAG pipelines operate as black boxes: retrieved documents influence outputs invisibly, preventing users, auditors, and regulators from verifying factual claims or detecting errors. This opacity constitutes a structural barrier to the auditability requirements embedded in fairness-oriented AI governance frameworks such as the EU AI Act. We introduce Interpretable Retrieval-Augmented Generation (I-RAG), a modular post-hoc framework that decomposes generated answers into atomic verifiable claims and bidirectionally links each claim to supporting evidence spans in source documents. Evaluated on 200 samples from Natural Questions and HotpotQA, I-RAG achieves 89.2% evidence support confidence on factual queries, 57.8% on complex multi-hop queries, and 98% overall claim coverage, with Attribution F1 of 0.693 outperforming sentence-level attribution baselines. All statistical comparisons against baselines are significant (p < 0.001 ). We additionally introduce NLI-based entailment scoring as a complementary attribution metric, addressing the inherent limitations of cosine-similarity-only evaluation. By making individual factual assertions traceable to their evidentiary basis, I-RAG provides a practical mechanism for transparency, human oversight, and accountability in AI-generated information systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v350-yesmin26a, title = {Beyond the Black Box: Claim-Level Interpretability for Accountable Retrieval-Augmented Generation}, author = {Yesmin, Farjana and Shirmin, Nusrat}, booktitle = {Proceedings of Fifth European Conference on Algorithmic Fairness}, pages = {165--177}, year = {2026}, editor = {De Bie, Tijl and Defrance, MaryBeth and Calders, Toon and Nguyen, Dennis}, volume = {350}, series = {Proceedings of Machine Learning Research}, month = {02--04 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v350/main/assets/yesmin26a/yesmin26a.pdf}, url = {https://proceedings.mlr.press/v350/yesmin26a.html}, abstract = {Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains such as healthcare, legal aid, and public information services contexts where opacity in AI decision-support raises fundamental accountability and fairness concerns. Standard RAG pipelines operate as black boxes: retrieved documents influence outputs invisibly, preventing users, auditors, and regulators from verifying factual claims or detecting errors. This opacity constitutes a structural barrier to the auditability requirements embedded in fairness-oriented AI governance frameworks such as the EU AI Act. We introduce Interpretable Retrieval-Augmented Generation (I-RAG), a modular post-hoc framework that decomposes generated answers into atomic verifiable claims and bidirectionally links each claim to supporting evidence spans in source documents. Evaluated on 200 samples from Natural Questions and HotpotQA, I-RAG achieves 89.2% evidence support confidence on factual queries, 57.8% on complex multi-hop queries, and 98% overall claim coverage, with Attribution F1 of 0.693 outperforming sentence-level attribution baselines. All statistical comparisons against baselines are significant (p < 0.001 ). We additionally introduce NLI-based entailment scoring as a complementary attribution metric, addressing the inherent limitations of cosine-similarity-only evaluation. By making individual factual assertions traceable to their evidentiary basis, I-RAG provides a practical mechanism for transparency, human oversight, and accountability in AI-generated information systems.} }
Endnote
%0 Conference Paper %T Beyond the Black Box: Claim-Level Interpretability for Accountable Retrieval-Augmented Generation %A Farjana Yesmin %A Nusrat Shirmin %B Proceedings of Fifth European Conference on Algorithmic Fairness %C Proceedings of Machine Learning Research %D 2026 %E Tijl De Bie %E MaryBeth Defrance %E Toon Calders %E Dennis Nguyen %F pmlr-v350-yesmin26a %I PMLR %P 165--177 %U https://proceedings.mlr.press/v350/yesmin26a.html %V 350 %X Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains such as healthcare, legal aid, and public information services contexts where opacity in AI decision-support raises fundamental accountability and fairness concerns. Standard RAG pipelines operate as black boxes: retrieved documents influence outputs invisibly, preventing users, auditors, and regulators from verifying factual claims or detecting errors. This opacity constitutes a structural barrier to the auditability requirements embedded in fairness-oriented AI governance frameworks such as the EU AI Act. We introduce Interpretable Retrieval-Augmented Generation (I-RAG), a modular post-hoc framework that decomposes generated answers into atomic verifiable claims and bidirectionally links each claim to supporting evidence spans in source documents. Evaluated on 200 samples from Natural Questions and HotpotQA, I-RAG achieves 89.2% evidence support confidence on factual queries, 57.8% on complex multi-hop queries, and 98% overall claim coverage, with Attribution F1 of 0.693 outperforming sentence-level attribution baselines. All statistical comparisons against baselines are significant (p < 0.001 ). We additionally introduce NLI-based entailment scoring as a complementary attribution metric, addressing the inherent limitations of cosine-similarity-only evaluation. By making individual factual assertions traceable to their evidentiary basis, I-RAG provides a practical mechanism for transparency, human oversight, and accountability in AI-generated information systems.
APA
Yesmin, F. & Shirmin, N.. (2026). Beyond the Black Box: Claim-Level Interpretability for Accountable Retrieval-Augmented Generation. Proceedings of Fifth European Conference on Algorithmic Fairness, in Proceedings of Machine Learning Research 350:165-177 Available from https://proceedings.mlr.press/v350/yesmin26a.html.

Related Material