Conformal LLM Multi-label Text Classification with Binary Relevance Approach

Viktor Örnbratt, Johan Hallberg Szabadváry
Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:214-229, 2025.

Abstract

Large Language Models (LLMs) are increasingly deployed in real-world Natural Language Processing (NLP) systems to perform multi-label classification tasks, such as identifying multiple forms of toxicity in online content. However, most models output raw probabilities without an exact way to quantify uncertainty, increasing the risk of over-prediction in high-stakes applications. In this work, we integrate Inductive Conformal Prediction (ICP) with the Binary Relevance (BR) approach to produce statistically valid prediction sets, label-wise. Using a modified Wikipedia Toxic Comments dataset, we evaluate this framework across varying significance levels ($\epsilon$), incorporating calibration-set-aware thresholds to address label imbalances. Our results show that BR-based conformal prediction maintains valid marginal coverage while enabling flexible control over prediction set size (efficiency). Even in the presence of rare labels, the framework provides practical uncertainty estimates and where the prediction can be abstained in uncertain cases via empty sets. These findings support the feasibility of BR-ICP-based uncertainty calibration for scalable, interpretable automation in multi-label NLP systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v266-ornbratt25a, title = {Conformal LLM Multi-label Text Classification with Binary Relevance Approach}, author = {\"{O}rnbratt, Viktor and Hallberg Szabadv\'{a}ry, Johan}, booktitle = {Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {214--229}, year = {2025}, editor = {Nguyen, Khuong An and Luo, Zhiyuan and Papadopoulos, Harris and Löfström, Tuwe and Carlsson, Lars and Boström, Henrik}, volume = {266}, series = {Proceedings of Machine Learning Research}, month = {10--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v266/main/assets/ornbratt25a/ornbratt25a.pdf}, url = {https://proceedings.mlr.press/v266/ornbratt25a.html}, abstract = {Large Language Models (LLMs) are increasingly deployed in real-world Natural Language Processing (NLP) systems to perform multi-label classification tasks, such as identifying multiple forms of toxicity in online content. However, most models output raw probabilities without an exact way to quantify uncertainty, increasing the risk of over-prediction in high-stakes applications. In this work, we integrate Inductive Conformal Prediction (ICP) with the Binary Relevance (BR) approach to produce statistically valid prediction sets, label-wise. Using a modified Wikipedia Toxic Comments dataset, we evaluate this framework across varying significance levels ($\epsilon$), incorporating calibration-set-aware thresholds to address label imbalances. Our results show that BR-based conformal prediction maintains valid marginal coverage while enabling flexible control over prediction set size (efficiency). Even in the presence of rare labels, the framework provides practical uncertainty estimates and where the prediction can be abstained in uncertain cases via empty sets. These findings support the feasibility of BR-ICP-based uncertainty calibration for scalable, interpretable automation in multi-label NLP systems.} }
Endnote
%0 Conference Paper %T Conformal LLM Multi-label Text Classification with Binary Relevance Approach %A Viktor Örnbratt %A Johan Hallberg Szabadváry %B Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2025 %E Khuong An Nguyen %E Zhiyuan Luo %E Harris Papadopoulos %E Tuwe Löfström %E Lars Carlsson %E Henrik Boström %F pmlr-v266-ornbratt25a %I PMLR %P 214--229 %U https://proceedings.mlr.press/v266/ornbratt25a.html %V 266 %X Large Language Models (LLMs) are increasingly deployed in real-world Natural Language Processing (NLP) systems to perform multi-label classification tasks, such as identifying multiple forms of toxicity in online content. However, most models output raw probabilities without an exact way to quantify uncertainty, increasing the risk of over-prediction in high-stakes applications. In this work, we integrate Inductive Conformal Prediction (ICP) with the Binary Relevance (BR) approach to produce statistically valid prediction sets, label-wise. Using a modified Wikipedia Toxic Comments dataset, we evaluate this framework across varying significance levels ($\epsilon$), incorporating calibration-set-aware thresholds to address label imbalances. Our results show that BR-based conformal prediction maintains valid marginal coverage while enabling flexible control over prediction set size (efficiency). Even in the presence of rare labels, the framework provides practical uncertainty estimates and where the prediction can be abstained in uncertain cases via empty sets. These findings support the feasibility of BR-ICP-based uncertainty calibration for scalable, interpretable automation in multi-label NLP systems.
APA
Örnbratt, V. & Hallberg Szabadváry, J.. (2025). Conformal LLM Multi-label Text Classification with Binary Relevance Approach. Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 266:214-229 Available from https://proceedings.mlr.press/v266/ornbratt25a.html.

Related Material