Optimal Number of Choices in Rating Contexts

Sam Ganzfried
Proceedings of the NIPS 2016 Workshop on Imperfect Decision Makers, PMLR 58:61-74, 2017.

Abstract

In many settings people must give numerical scores to entities from a small discrete set. For instance, rating physical attractiveness from 1–5 on dating sites, or papers from 1–10 for conference reviewing. We study the problem of understanding when using a different number of options is optimal. For concreteness we assume the true underlying scores are integers from 1–100. We consider the case when scores are uniform random and Gaussian. While in theory for this setting it would be optimal to use all 100 options, in practice this is prohibitive, and it is preferable to utilize a smaller number of options due to humans’ cognitive limitations. Our results suggest that using a smaller number of options than is typical could be optimal in certain situations. This would have many potential applications, as settings requiring entities to be ranked by humans are ubiquitous.

Cite this Paper

BibTeX
@InProceedings{pmlr-v58-ganzfried17a, title = {Optimal Number of Choices in Rating Contexts}, author = {Ganzfried, Sam}, booktitle = {Proceedings of the NIPS 2016 Workshop on Imperfect Decision Makers}, pages = {61--74}, year = {2017}, editor = {Guy, Tatiana V. and Kárný, Miroslav and Rios-Insua, David and Wolpert, David H.}, volume = {58}, series = {Proceedings of Machine Learning Research}, month = {09 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v58/ganzfried17a/ganzfried17a.pdf}, url = {https://proceedings.mlr.press/v58/ganzfried17a.html}, abstract = {In many settings people must give numerical scores to entities from a small discrete set. For instance, rating physical attractiveness from 1–5 on dating sites, or papers from 1–10 for conference reviewing. We study the problem of understanding when using a different number of options is optimal. For concreteness we assume the true underlying scores are integers from 1–100. We consider the case when scores are uniform random and Gaussian. While in theory for this setting it would be optimal to use all 100 options, in practice this is prohibitive, and it is preferable to utilize a smaller number of options due to humans’ cognitive limitations. Our results suggest that using a smaller number of options than is typical could be optimal in certain situations. This would have many potential applications, as settings requiring entities to be ranked by humans are ubiquitous.} }
Endnote
%0 Conference Paper %T Optimal Number of Choices in Rating Contexts %A Sam Ganzfried %B Proceedings of the NIPS 2016 Workshop on Imperfect Decision Makers %C Proceedings of Machine Learning Research %D 2017 %E Tatiana V. Guy %E Miroslav Kárný %E David Rios-Insua %E David H. Wolpert %F pmlr-v58-ganzfried17a %I PMLR %P 61--74 %U https://proceedings.mlr.press/v58/ganzfried17a.html %V 58 %X In many settings people must give numerical scores to entities from a small discrete set. For instance, rating physical attractiveness from 1–5 on dating sites, or papers from 1–10 for conference reviewing. We study the problem of understanding when using a different number of options is optimal. For concreteness we assume the true underlying scores are integers from 1–100. We consider the case when scores are uniform random and Gaussian. While in theory for this setting it would be optimal to use all 100 options, in practice this is prohibitive, and it is preferable to utilize a smaller number of options due to humans’ cognitive limitations. Our results suggest that using a smaller number of options than is typical could be optimal in certain situations. This would have many potential applications, as settings requiring entities to be ranked by humans are ubiquitous.
APA
Ganzfried, S.. (2017). Optimal Number of Choices in Rating Contexts. Proceedings of the NIPS 2016 Workshop on Imperfect Decision Makers, in Proceedings of Machine Learning Research 58:61-74 Available from https://proceedings.mlr.press/v58/ganzfried17a.html.

Related Material