A Strategy for Ranking Optimization Methods using Multiple Criteria

Ian Dewancker, Michael McCourt, Scott Clark, Patrick Hayes, Alexandra Johnson, George Ke
Proceedings of the Workshop on Automatic Machine Learning, PMLR 64:11-20, 2016.

Abstract

Many methods for optimizing black-box functions exist, and many metrics exist for judging the performance of a specific optimization method. There is not, however, a generally agreed upon strategy for simultaneously comparing the performance of multiple optimization methods for multiple performance metrics across a range of optimization problems. This paper proposes such a methodology, which uses nonparametric statistical tests to convert the metrics recorded for each problem into a partial ranking of optimization methods; these partial rankings are then amalgamated through a voting mechanism to generate a final score for each optimization method. Mathematical analysis is provided to motivate decisions within this strategy, and numerical results are provided to demonstrate the potential insights afforded thereby.

Cite this Paper

BibTeX
@InProceedings{pmlr-v64-dewancker_strategy_2016, title = {A Strategy for Ranking Optimization Methods using Multiple Criteria}, author = {Dewancker, Ian and McCourt, Michael and Clark, Scott and Hayes, Patrick and Johnson, Alexandra and Ke, George}, booktitle = {Proceedings of the Workshop on Automatic Machine Learning}, pages = {11--20}, year = {2016}, editor = {Hutter, Frank and Kotthoff, Lars and Vanschoren, Joaquin}, volume = {64}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf}, url = {https://proceedings.mlr.press/v64/dewancker_strategy_2016.html}, abstract = {Many methods for optimizing black-box functions exist, and many metrics exist for judging the performance of a specific optimization method. There is not, however, a generally agreed upon strategy for simultaneously comparing the performance of multiple optimization methods for multiple performance metrics across a range of optimization problems. This paper proposes such a methodology, which uses nonparametric statistical tests to convert the metrics recorded for each problem into a partial ranking of optimization methods; these partial rankings are then amalgamated through a voting mechanism to generate a final score for each optimization method. Mathematical analysis is provided to motivate decisions within this strategy, and numerical results are provided to demonstrate the potential insights afforded thereby.} }
Endnote
%0 Conference Paper %T A Strategy for Ranking Optimization Methods using Multiple Criteria %A Ian Dewancker %A Michael McCourt %A Scott Clark %A Patrick Hayes %A Alexandra Johnson %A George Ke %B Proceedings of the Workshop on Automatic Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Frank Hutter %E Lars Kotthoff %E Joaquin Vanschoren %F pmlr-v64-dewancker_strategy_2016 %I PMLR %P 11--20 %U https://proceedings.mlr.press/v64/dewancker_strategy_2016.html %V 64 %X Many methods for optimizing black-box functions exist, and many metrics exist for judging the performance of a specific optimization method. There is not, however, a generally agreed upon strategy for simultaneously comparing the performance of multiple optimization methods for multiple performance metrics across a range of optimization problems. This paper proposes such a methodology, which uses nonparametric statistical tests to convert the metrics recorded for each problem into a partial ranking of optimization methods; these partial rankings are then amalgamated through a voting mechanism to generate a final score for each optimization method. Mathematical analysis is provided to motivate decisions within this strategy, and numerical results are provided to demonstrate the potential insights afforded thereby.
RIS
TY - CPAPER TI - A Strategy for Ranking Optimization Methods using Multiple Criteria AU - Ian Dewancker AU - Michael McCourt AU - Scott Clark AU - Patrick Hayes AU - Alexandra Johnson AU - George Ke BT - Proceedings of the Workshop on Automatic Machine Learning DA - 2016/12/04 ED - Frank Hutter ED - Lars Kotthoff ED - Joaquin Vanschoren ID - pmlr-v64-dewancker_strategy_2016 PB - PMLR DP - Proceedings of Machine Learning Research VL - 64 SP - 11 EP - 20 L1 - http://proceedings.mlr.press/v64/dewancker_strategy_2016.pdf UR - https://proceedings.mlr.press/v64/dewancker_strategy_2016.html AB - Many methods for optimizing black-box functions exist, and many metrics exist for judging the performance of a specific optimization method. There is not, however, a generally agreed upon strategy for simultaneously comparing the performance of multiple optimization methods for multiple performance metrics across a range of optimization problems. This paper proposes such a methodology, which uses nonparametric statistical tests to convert the metrics recorded for each problem into a partial ranking of optimization methods; these partial rankings are then amalgamated through a voting mechanism to generate a final score for each optimization method. Mathematical analysis is provided to motivate decisions within this strategy, and numerical results are provided to demonstrate the potential insights afforded thereby. ER -
APA
Dewancker, I., McCourt, M., Clark, S., Hayes, P., Johnson, A. & Ke, G.. (2016). A Strategy for Ranking Optimization Methods using Multiple Criteria. Proceedings of the Workshop on Automatic Machine Learning, in Proceedings of Machine Learning Research 64:11-20 Available from https://proceedings.mlr.press/v64/dewancker_strategy_2016.html.

Related Material