A Deep Neural Network Conformal Predictor for Multi-label Text Classification

[edit]

Andreas Paisios, Ladislav Lenc, Jiřı́ Martı́nek, Pavel Král, Harris Papadopoulos ;
Proceedings of the Eighth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 105:228-245, 2019.

Abstract

We investigate the use of inductive conformal prediction (ICP) for the task of multi-label text classification and present preliminary experimental results for a subset of the original Reuters-21578 data-set. Our underlying classification model is a deep neural network configuration which consists of a trainable embedding layer, a convolutional layer and two dense feed-forward layers, arranged sequentially, with sigmoid outputs representing the individual unique labels of the selected subset. Following the power-set approach, we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined and we experiment with a number of different versions of a nonconformity measure. Our results indicate a good performance for the underlying model which is carried on to the ICP without any significant accuracy loss and with the added benefits of prediction-specific confidence information. Prediction-sets are tight enough to be practically useful even though the multi-label subset contains tens of thousands of possible label combinations and empirical error-rates confirm that our outputs are well-calibrated.

Related Material