Scaling of Class-wise Training Losses for Post-hoc Calibration

Seungjin Jung, Seungmo Seo, Yonghyun Jeong, Jongwon Choi
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:15421-15434, 2023.

Abstract

The class-wise training losses often diverge as a result of the various levels of intra-class and inter-class appearance variation, and we find that the diverging class-wise training losses cause the uncalibrated prediction with its reliability. To resolve the issue, we propose a new calibration method to synchronize the class-wise training losses. We design a new training loss to alleviate the variance of class-wise training losses by using multiple class-wise scaling factors. Since our framework can compensate the training losses of overfitted classes with those of under-fitted classes, the integrated training loss is preserved, preventing the performance drop even after the model calibration. Furthermore, our method can be easily employed in the post-hoc calibration methods, allowing us to use the pre-trained model as an initial model and reduce the additional computation for model calibration. We validate the proposed framework by employing it in the various post-hoc calibration methods, which generally improves calibration performance while preserving accuracy, and discover through the investigation that our approach performs well with unbalanced datasets and untuned hyperparameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-jung23a, title = {Scaling of Class-wise Training Losses for Post-hoc Calibration}, author = {Jung, Seungjin and Seo, Seungmo and Jeong, Yonghyun and Choi, Jongwon}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {15421--15434}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/jung23a/jung23a.pdf}, url = {https://proceedings.mlr.press/v202/jung23a.html}, abstract = {The class-wise training losses often diverge as a result of the various levels of intra-class and inter-class appearance variation, and we find that the diverging class-wise training losses cause the uncalibrated prediction with its reliability. To resolve the issue, we propose a new calibration method to synchronize the class-wise training losses. We design a new training loss to alleviate the variance of class-wise training losses by using multiple class-wise scaling factors. Since our framework can compensate the training losses of overfitted classes with those of under-fitted classes, the integrated training loss is preserved, preventing the performance drop even after the model calibration. Furthermore, our method can be easily employed in the post-hoc calibration methods, allowing us to use the pre-trained model as an initial model and reduce the additional computation for model calibration. We validate the proposed framework by employing it in the various post-hoc calibration methods, which generally improves calibration performance while preserving accuracy, and discover through the investigation that our approach performs well with unbalanced datasets and untuned hyperparameters.} }
Endnote
%0 Conference Paper %T Scaling of Class-wise Training Losses for Post-hoc Calibration %A Seungjin Jung %A Seungmo Seo %A Yonghyun Jeong %A Jongwon Choi %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-jung23a %I PMLR %P 15421--15434 %U https://proceedings.mlr.press/v202/jung23a.html %V 202 %X The class-wise training losses often diverge as a result of the various levels of intra-class and inter-class appearance variation, and we find that the diverging class-wise training losses cause the uncalibrated prediction with its reliability. To resolve the issue, we propose a new calibration method to synchronize the class-wise training losses. We design a new training loss to alleviate the variance of class-wise training losses by using multiple class-wise scaling factors. Since our framework can compensate the training losses of overfitted classes with those of under-fitted classes, the integrated training loss is preserved, preventing the performance drop even after the model calibration. Furthermore, our method can be easily employed in the post-hoc calibration methods, allowing us to use the pre-trained model as an initial model and reduce the additional computation for model calibration. We validate the proposed framework by employing it in the various post-hoc calibration methods, which generally improves calibration performance while preserving accuracy, and discover through the investigation that our approach performs well with unbalanced datasets and untuned hyperparameters.
APA
Jung, S., Seo, S., Jeong, Y. & Choi, J.. (2023). Scaling of Class-wise Training Losses for Post-hoc Calibration. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:15421-15434 Available from https://proceedings.mlr.press/v202/jung23a.html.

Related Material