What is the Right Notion of Distance between Predict-then-Optimize Tasks?

Paula Rodriguez-Diaz, Lingkai Kong, Kai Wang, David Alvarez-Melis, Milind Tambe
Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, PMLR 286:3570-3586, 2025.

Abstract

Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms-from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret rather than prediction error. In this work, we propose OTD$^3$ (Optimal Transport Decision-aware Dataset Distance), a novel dataset distance that incorporates downstream decisions in addition to features and labels. We show that traditional feature-label distances lack informativeness in PtO settings, while OTD$^3$ more effectively captures adaptation success. We also derive a PtO-specific adaptation bound based on this distance. Empirically, we show that our proposed distance accurately predicts model transferability across three different PtO tasks from the literature. Code is available at https://github.com/paularodr/OTD3

Cite this Paper

BibTeX
@InProceedings{pmlr-v286-rodriguez-diaz25a, title = {What is the Right Notion of Distance between Predict-then-Optimize Tasks?}, author = {Rodriguez-Diaz, Paula and Kong, Lingkai and Wang, Kai and Alvarez-Melis, David and Tambe, Milind}, booktitle = {Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence}, pages = {3570--3586}, year = {2025}, editor = {Chiappa, Silvia and Magliacane, Sara}, volume = {286}, series = {Proceedings of Machine Learning Research}, month = {21--25 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v286/main/assets/rodriguez-diaz25a/rodriguez-diaz25a.pdf}, url = {https://proceedings.mlr.press/v286/rodriguez-diaz25a.html}, abstract = {Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms-from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret rather than prediction error. In this work, we propose OTD$^3$ (Optimal Transport Decision-aware Dataset Distance), a novel dataset distance that incorporates downstream decisions in addition to features and labels. We show that traditional feature-label distances lack informativeness in PtO settings, while OTD$^3$ more effectively captures adaptation success. We also derive a PtO-specific adaptation bound based on this distance. Empirically, we show that our proposed distance accurately predicts model transferability across three different PtO tasks from the literature. Code is available at https://github.com/paularodr/OTD3} }
Endnote
%0 Conference Paper %T What is the Right Notion of Distance between Predict-then-Optimize Tasks? %A Paula Rodriguez-Diaz %A Lingkai Kong %A Kai Wang %A David Alvarez-Melis %A Milind Tambe %B Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2025 %E Silvia Chiappa %E Sara Magliacane %F pmlr-v286-rodriguez-diaz25a %I PMLR %P 3570--3586 %U https://proceedings.mlr.press/v286/rodriguez-diaz25a.html %V 286 %X Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms-from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret rather than prediction error. In this work, we propose OTD$^3$ (Optimal Transport Decision-aware Dataset Distance), a novel dataset distance that incorporates downstream decisions in addition to features and labels. We show that traditional feature-label distances lack informativeness in PtO settings, while OTD$^3$ more effectively captures adaptation success. We also derive a PtO-specific adaptation bound based on this distance. Empirically, we show that our proposed distance accurately predicts model transferability across three different PtO tasks from the literature. Code is available at https://github.com/paularodr/OTD3
APA
Rodriguez-Diaz, P., Kong, L., Wang, K., Alvarez-Melis, D. & Tambe, M.. (2025). What is the Right Notion of Distance between Predict-then-Optimize Tasks?. Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 286:3570-3586 Available from https://proceedings.mlr.press/v286/rodriguez-diaz25a.html.

Related Material