Exploiting Performance-based Similarity between Datasets in Metalearning

Rui Leite, Pavel Brazdil
AAAI Workshop on Meta-Learning and MetaDL Challenge, PMLR 140:90-99, 2021.

Abstract

This paper describes an improved algorithm selection method of a previous method called active testing. This method seeks a workflow (or its particular configuration) that would lead to the highest gain in performance (e.g., accuracy). The new version uses a particular performance-based characterization of each dataset, which is in the form of a vector of performance values of different algorithms. Dataset similarity is then assessed by comparing these performance vectors. One useful measure for this comparison is Spearman’s correlation. The advantage of this measure is that it can be easily recalculated as more information is gathered. Consequently, as the tests proceed, the recommendations of the system get adjusted to the characteristics of the target dataset. We show that this new strategy leads to improved results of the active testing approach.

Cite this Paper

BibTeX
@InProceedings{pmlr-v140-leite21a, title = {Exploiting Performance-based Similarity between Datasets in Metalearning}, author = {Leite, Rui and Brazdil, Pavel}, booktitle = {AAAI Workshop on Meta-Learning and MetaDL Challenge}, pages = {90--99}, year = {2021}, editor = {Guyon, Isabelle and van Rijn, Jan N. and Treguer, Sébastien and Vanschoren, Joaquin}, volume = {140}, series = {Proceedings of Machine Learning Research}, month = {09 Feb}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v140/leite21a/leite21a.pdf}, url = {https://proceedings.mlr.press/v140/leite21a.html}, abstract = {This paper describes an improved algorithm selection method of a previous method called active testing. This method seeks a workflow (or its particular configuration) that would lead to the highest gain in performance (e.g., accuracy). The new version uses a particular performance-based characterization of each dataset, which is in the form of a vector of performance values of different algorithms. Dataset similarity is then assessed by comparing these performance vectors. One useful measure for this comparison is Spearman’s correlation. The advantage of this measure is that it can be easily recalculated as more information is gathered. Consequently, as the tests proceed, the recommendations of the system get adjusted to the characteristics of the target dataset. We show that this new strategy leads to improved results of the active testing approach.} }
Endnote
%0 Conference Paper %T Exploiting Performance-based Similarity between Datasets in Metalearning %A Rui Leite %A Pavel Brazdil %B AAAI Workshop on Meta-Learning and MetaDL Challenge %C Proceedings of Machine Learning Research %D 2021 %E Isabelle Guyon %E Jan N. van Rijn %E Sébastien Treguer %E Joaquin Vanschoren %F pmlr-v140-leite21a %I PMLR %P 90--99 %U https://proceedings.mlr.press/v140/leite21a.html %V 140 %X This paper describes an improved algorithm selection method of a previous method called active testing. This method seeks a workflow (or its particular configuration) that would lead to the highest gain in performance (e.g., accuracy). The new version uses a particular performance-based characterization of each dataset, which is in the form of a vector of performance values of different algorithms. Dataset similarity is then assessed by comparing these performance vectors. One useful measure for this comparison is Spearman’s correlation. The advantage of this measure is that it can be easily recalculated as more information is gathered. Consequently, as the tests proceed, the recommendations of the system get adjusted to the characteristics of the target dataset. We show that this new strategy leads to improved results of the active testing approach.
APA
Leite, R. & Brazdil, P.. (2021). Exploiting Performance-based Similarity between Datasets in Metalearning. AAAI Workshop on Meta-Learning and MetaDL Challenge, in Proceedings of Machine Learning Research 140:90-99 Available from https://proceedings.mlr.press/v140/leite21a.html.

Related Material