Trees with Attention for Set Prediction Tasks

Roy Hirsch, Ran Gilad-Bachrach
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:4250-4261, 2021.

Abstract

In many machine learning applications, each record represents a set of items. For example, when making predictions from medical records, the medications prescribed to a patient are a set whose size is not fixed and whose order is arbitrary. However, most machine learning algorithms are not designed to handle set structures and are limited to processing records of fixed size. Set-Tree, presented in this work, extends the support for sets to tree-based models, such as Random-Forest and Gradient-Boosting, by introducing an attention mechanism and set-compatible split criteria. We evaluate the new method empirically on a wide range of problems ranging from making predictions on sub-atomic particle jets to estimating the redshift of galaxies. The new method outperforms existing tree-based methods consistently and significantly. Moreover, it is competitive and often outperforms Deep Learning. We also discuss the theoretical properties of Set-Trees and explain how they enable item-level explainability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-hirsch21a, title = {Trees with Attention for Set Prediction Tasks}, author = {Hirsch, Roy and Gilad-Bachrach, Ran}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {4250--4261}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/hirsch21a/hirsch21a.pdf}, url = {https://proceedings.mlr.press/v139/hirsch21a.html}, abstract = {In many machine learning applications, each record represents a set of items. For example, when making predictions from medical records, the medications prescribed to a patient are a set whose size is not fixed and whose order is arbitrary. However, most machine learning algorithms are not designed to handle set structures and are limited to processing records of fixed size. Set-Tree, presented in this work, extends the support for sets to tree-based models, such as Random-Forest and Gradient-Boosting, by introducing an attention mechanism and set-compatible split criteria. We evaluate the new method empirically on a wide range of problems ranging from making predictions on sub-atomic particle jets to estimating the redshift of galaxies. The new method outperforms existing tree-based methods consistently and significantly. Moreover, it is competitive and often outperforms Deep Learning. We also discuss the theoretical properties of Set-Trees and explain how they enable item-level explainability.} }
Endnote
%0 Conference Paper %T Trees with Attention for Set Prediction Tasks %A Roy Hirsch %A Ran Gilad-Bachrach %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-hirsch21a %I PMLR %P 4250--4261 %U https://proceedings.mlr.press/v139/hirsch21a.html %V 139 %X In many machine learning applications, each record represents a set of items. For example, when making predictions from medical records, the medications prescribed to a patient are a set whose size is not fixed and whose order is arbitrary. However, most machine learning algorithms are not designed to handle set structures and are limited to processing records of fixed size. Set-Tree, presented in this work, extends the support for sets to tree-based models, such as Random-Forest and Gradient-Boosting, by introducing an attention mechanism and set-compatible split criteria. We evaluate the new method empirically on a wide range of problems ranging from making predictions on sub-atomic particle jets to estimating the redshift of galaxies. The new method outperforms existing tree-based methods consistently and significantly. Moreover, it is competitive and often outperforms Deep Learning. We also discuss the theoretical properties of Set-Trees and explain how they enable item-level explainability.
APA
Hirsch, R. & Gilad-Bachrach, R.. (2021). Trees with Attention for Set Prediction Tasks. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:4250-4261 Available from https://proceedings.mlr.press/v139/hirsch21a.html.

Related Material