Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures

Martijn M Gösgens, Alexey Tikhonov, Liudmila Prokhorenkova
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:3799-3808, 2021.

Abstract

Many cluster similarity indices are used to evaluate clustering algorithms, and choosing the best one for a particular task remains an open problem. We demonstrate that this problem is crucial: there are many disagreements among the indices, these disagreements do affect which algorithms are preferred in applications, and this can lead to degraded performance in real-world systems. We propose a theoretical framework to tackle this problem: we develop a list of desirable properties and conduct an extensive theoretical analysis to verify which indices satisfy them. This allows for making an informed choice: given a particular application, one can first select properties that are desirable for the task and then identify indices satisfying these. Our work unifies and considerably extends existing attempts at analyzing cluster similarity indices: we introduce new properties, formalize existing ones, and mathematically prove or disprove each property for an extensive list of validation indices. This broader and more rigorous approach leads to recommendations that considerably differ from how validation indices are currently being chosen by practitioners. Some of the most popular indices are even shown to be dominated by previously overlooked ones.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-gosgens21a, title = {Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures}, author = {G{\"o}sgens, Martijn M and Tikhonov, Alexey and Prokhorenkova, Liudmila}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {3799--3808}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/gosgens21a/gosgens21a.pdf}, url = {https://proceedings.mlr.press/v139/gosgens21a.html}, abstract = {Many cluster similarity indices are used to evaluate clustering algorithms, and choosing the best one for a particular task remains an open problem. We demonstrate that this problem is crucial: there are many disagreements among the indices, these disagreements do affect which algorithms are preferred in applications, and this can lead to degraded performance in real-world systems. We propose a theoretical framework to tackle this problem: we develop a list of desirable properties and conduct an extensive theoretical analysis to verify which indices satisfy them. This allows for making an informed choice: given a particular application, one can first select properties that are desirable for the task and then identify indices satisfying these. Our work unifies and considerably extends existing attempts at analyzing cluster similarity indices: we introduce new properties, formalize existing ones, and mathematically prove or disprove each property for an extensive list of validation indices. This broader and more rigorous approach leads to recommendations that considerably differ from how validation indices are currently being chosen by practitioners. Some of the most popular indices are even shown to be dominated by previously overlooked ones.} }
Endnote
%0 Conference Paper %T Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures %A Martijn M Gösgens %A Alexey Tikhonov %A Liudmila Prokhorenkova %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-gosgens21a %I PMLR %P 3799--3808 %U https://proceedings.mlr.press/v139/gosgens21a.html %V 139 %X Many cluster similarity indices are used to evaluate clustering algorithms, and choosing the best one for a particular task remains an open problem. We demonstrate that this problem is crucial: there are many disagreements among the indices, these disagreements do affect which algorithms are preferred in applications, and this can lead to degraded performance in real-world systems. We propose a theoretical framework to tackle this problem: we develop a list of desirable properties and conduct an extensive theoretical analysis to verify which indices satisfy them. This allows for making an informed choice: given a particular application, one can first select properties that are desirable for the task and then identify indices satisfying these. Our work unifies and considerably extends existing attempts at analyzing cluster similarity indices: we introduce new properties, formalize existing ones, and mathematically prove or disprove each property for an extensive list of validation indices. This broader and more rigorous approach leads to recommendations that considerably differ from how validation indices are currently being chosen by practitioners. Some of the most popular indices are even shown to be dominated by previously overlooked ones.
APA
Gösgens, M.M., Tikhonov, A. & Prokhorenkova, L.. (2021). Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:3799-3808 Available from https://proceedings.mlr.press/v139/gosgens21a.html.

Related Material