A Deep Neural Network Conformal Predictor for Multi-label Text Classification

Andreas Paisios, Ladislav Lenc, Jiřı́ Martı́nek, Pavel Král, Harris Papadopoulos
Proceedings of the Eighth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 105:228-245, 2019.

Abstract

We investigate the use of inductive conformal prediction (ICP) for the task of multi-label text classification and present preliminary experimental results for a subset of the original Reuters-21578 data-set. Our underlying classification model is a deep neural network configuration which consists of a trainable embedding layer, a convolutional layer and two dense feed-forward layers, arranged sequentially, with sigmoid outputs representing the individual unique labels of the selected subset. Following the power-set approach, we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined and we experiment with a number of different versions of a nonconformity measure. Our results indicate a good performance for the underlying model which is carried on to the ICP without any significant accuracy loss and with the added benefits of prediction-specific confidence information. Prediction-sets are tight enough to be practically useful even though the multi-label subset contains tens of thousands of possible label combinations and empirical error-rates confirm that our outputs are well-calibrated.

Cite this Paper

BibTeX
@InProceedings{pmlr-v105-paisios19a, title = {A Deep Neural Network Conformal Predictor for Multi-label Text Classification}, author = {Paisios, Andreas and Lenc, Ladislav and Mart\'{\i}nek, Ji\v{r}\'{\i} and Kr\'al, Pavel and Papadopoulos, Harris}, booktitle = {Proceedings of the Eighth Symposium on Conformal and Probabilistic Prediction and Applications}, pages = {228--245}, year = {2019}, editor = {Gammerman, Alex and Vovk, Vladimir and Luo, Zhiyuan and Smirnov, Evgueni}, volume = {105}, series = {Proceedings of Machine Learning Research}, month = {09--11 Sep}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v105/paisios19a/paisios19a.pdf}, url = {https://proceedings.mlr.press/v105/paisios19a.html}, abstract = {We investigate the use of inductive conformal prediction (ICP) for the task of multi-label text classification and present preliminary experimental results for a subset of the original Reuters-21578 data-set. Our underlying classification model is a deep neural network configuration which consists of a trainable embedding layer, a convolutional layer and two dense feed-forward layers, arranged sequentially, with sigmoid outputs representing the individual unique labels of the selected subset. Following the power-set approach, we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined and we experiment with a number of different versions of a nonconformity measure. Our results indicate a good performance for the underlying model which is carried on to the ICP without any significant accuracy loss and with the added benefits of prediction-specific confidence information. Prediction-sets are tight enough to be practically useful even though the multi-label subset contains tens of thousands of possible label combinations and empirical error-rates confirm that our outputs are well-calibrated.} }
Endnote
%0 Conference Paper %T A Deep Neural Network Conformal Predictor for Multi-label Text Classification %A Andreas Paisios %A Ladislav Lenc %A Jiřı́ Martı́nek %A Pavel Král %A Harris Papadopoulos %B Proceedings of the Eighth Symposium on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2019 %E Alex Gammerman %E Vladimir Vovk %E Zhiyuan Luo %E Evgueni Smirnov %F pmlr-v105-paisios19a %I PMLR %P 228--245 %U https://proceedings.mlr.press/v105/paisios19a.html %V 105 %X We investigate the use of inductive conformal prediction (ICP) for the task of multi-label text classification and present preliminary experimental results for a subset of the original Reuters-21578 data-set. Our underlying classification model is a deep neural network configuration which consists of a trainable embedding layer, a convolutional layer and two dense feed-forward layers, arranged sequentially, with sigmoid outputs representing the individual unique labels of the selected subset. Following the power-set approach, we assign nonconformity scores to label-sets from which the corresponding p-values and prediction-sets are determined and we experiment with a number of different versions of a nonconformity measure. Our results indicate a good performance for the underlying model which is carried on to the ICP without any significant accuracy loss and with the added benefits of prediction-specific confidence information. Prediction-sets are tight enough to be practically useful even though the multi-label subset contains tens of thousands of possible label combinations and empirical error-rates confirm that our outputs are well-calibrated.
APA
Paisios, A., Lenc, L., Martı́nek, J., Král, P. & Papadopoulos, H.. (2019). A Deep Neural Network Conformal Predictor for Multi-label Text Classification. Proceedings of the Eighth Symposium on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 105:228-245 Available from https://proceedings.mlr.press/v105/paisios19a.html.

Related Material