PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

Ivan Stelmakh, Nihar B. Shah, Aarti Singh
Proceedings of the 30th International Conference on Algorithmic Learning Theory, PMLR 98:828-856, 2019.

Abstract

We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the popular objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. With a sharp minimax analysis we also prove that our algorithm leads to assignments with strong statistical guarantees both in an objective-score model as well as a novel subjective-score model that we propose in this paper.

Cite this Paper

BibTeX
@InProceedings{pmlr-v98-stelmakh19a, title = {PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review}, author = {Stelmakh, Ivan and Shah, Nihar B. and Singh, Aarti}, booktitle = {Proceedings of the 30th International Conference on Algorithmic Learning Theory}, pages = {828--856}, year = {2019}, editor = {Garivier, Aurélien and Kale, Satyen}, volume = {98}, series = {Proceedings of Machine Learning Research}, month = {22--24 Mar}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v98/stelmakh19a/stelmakh19a.pdf}, url = {https://proceedings.mlr.press/v98/stelmakh19a.html}, abstract = {We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the popular objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. With a sharp minimax analysis we also prove that our algorithm leads to assignments with strong statistical guarantees both in an objective-score model as well as a novel subjective-score model that we propose in this paper.} }
Endnote
%0 Conference Paper %T PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review %A Ivan Stelmakh %A Nihar B. Shah %A Aarti Singh %B Proceedings of the 30th International Conference on Algorithmic Learning Theory %C Proceedings of Machine Learning Research %D 2019 %E Aurélien Garivier %E Satyen Kale %F pmlr-v98-stelmakh19a %I PMLR %P 828--856 %U https://proceedings.mlr.press/v98/stelmakh19a.html %V 98 %X We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the popular objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. With a sharp minimax analysis we also prove that our algorithm leads to assignments with strong statistical guarantees both in an objective-score model as well as a novel subjective-score model that we propose in this paper.
APA
Stelmakh, I., Shah, N.B. & Singh, A.. (2019). PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review. Proceedings of the 30th International Conference on Algorithmic Learning Theory, in Proceedings of Machine Learning Research 98:828-856 Available from https://proceedings.mlr.press/v98/stelmakh19a.html.

Related Material