$β^3$-IRT: A New Item Response Model and its Applications

Yu Chen, Telmo Silva Filho, Ricardo B. Prudencio, Tom Diethe, Peter Flach
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:1013-1021, 2019.

Abstract

Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $\beta^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves. In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $\beta^3$-IRT to assess the ability of machine learning classifiers.This novel application results in a new metric for evaluating the quality of the classifier’s probability estimates, based on the inferred difficulty and discrimination of data instances.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-chen19b, title = {$β^3$-IRT: A New Item Response Model and its Applications}, author = {Chen, Yu and Filho, Telmo Silva and Prudencio, Ricardo B. and Diethe, Tom and Flach, Peter}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {1013--1021}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/chen19b/chen19b.pdf}, url = {https://proceedings.mlr.press/v89/chen19b.html}, abstract = {Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $\beta^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves. In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $\beta^3$-IRT to assess the ability of machine learning classifiers.This novel application results in a new metric for evaluating the quality of the classifier’s probability estimates, based on the inferred difficulty and discrimination of data instances.} }
Endnote
%0 Conference Paper %T $β^3$-IRT: A New Item Response Model and its Applications %A Yu Chen %A Telmo Silva Filho %A Ricardo B. Prudencio %A Tom Diethe %A Peter Flach %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-chen19b %I PMLR %P 1013--1021 %U https://proceedings.mlr.press/v89/chen19b.html %V 89 %X Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the $\beta^3$-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves. In experiments we applied the proposed model to data from an online exam platform, and show our model outperforms a more standard 2PL-ND model on all datasets. Furthermore, we show how to apply $\beta^3$-IRT to assess the ability of machine learning classifiers.This novel application results in a new metric for evaluating the quality of the classifier’s probability estimates, based on the inferred difficulty and discrimination of data instances.
APA
Chen, Y., Filho, T.S., Prudencio, R.B., Diethe, T. & Flach, P.. (2019). $β^3$-IRT: A New Item Response Model and its Applications. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:1013-1021 Available from https://proceedings.mlr.press/v89/chen19b.html.

Related Material