Evaluation of Trajectory Distribution Predictions with Energy Score

Novin Shahroudi, Mihkel Lepson, Meelis Kull
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:44322-44341, 2024.

Abstract

Predicting the future trajectory of surrounding objects is inherently uncertain and vital in the safe and reliable planning of autonomous systems such as in self-driving cars. Although trajectory prediction models have become increasingly sophisticated in dealing with the complexities of spatiotemporal data, the evaluation methods used to assess these models have not kept pace. "Minimum of N" is a common family of metrics used to assess the rich outputs of such models. We critically examine the Minimum of N within the proper scoring rules framework to show that it is not strictly proper and demonstrate how that could lead to a misleading assessment of multimodal trajectory predictions. As an alternative, we propose using Energy Score-based evaluation measures, leveraging their proven propriety for a more reliable evaluation of trajectory distribution predictions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-shahroudi24a, title = {Evaluation of Trajectory Distribution Predictions with Energy Score}, author = {Shahroudi, Novin and Lepson, Mihkel and Kull, Meelis}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {44322--44341}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/shahroudi24a/shahroudi24a.pdf}, url = {https://proceedings.mlr.press/v235/shahroudi24a.html}, abstract = {Predicting the future trajectory of surrounding objects is inherently uncertain and vital in the safe and reliable planning of autonomous systems such as in self-driving cars. Although trajectory prediction models have become increasingly sophisticated in dealing with the complexities of spatiotemporal data, the evaluation methods used to assess these models have not kept pace. "Minimum of N" is a common family of metrics used to assess the rich outputs of such models. We critically examine the Minimum of N within the proper scoring rules framework to show that it is not strictly proper and demonstrate how that could lead to a misleading assessment of multimodal trajectory predictions. As an alternative, we propose using Energy Score-based evaluation measures, leveraging their proven propriety for a more reliable evaluation of trajectory distribution predictions.} }
Endnote
%0 Conference Paper %T Evaluation of Trajectory Distribution Predictions with Energy Score %A Novin Shahroudi %A Mihkel Lepson %A Meelis Kull %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-shahroudi24a %I PMLR %P 44322--44341 %U https://proceedings.mlr.press/v235/shahroudi24a.html %V 235 %X Predicting the future trajectory of surrounding objects is inherently uncertain and vital in the safe and reliable planning of autonomous systems such as in self-driving cars. Although trajectory prediction models have become increasingly sophisticated in dealing with the complexities of spatiotemporal data, the evaluation methods used to assess these models have not kept pace. "Minimum of N" is a common family of metrics used to assess the rich outputs of such models. We critically examine the Minimum of N within the proper scoring rules framework to show that it is not strictly proper and demonstrate how that could lead to a misleading assessment of multimodal trajectory predictions. As an alternative, we propose using Energy Score-based evaluation measures, leveraging their proven propriety for a more reliable evaluation of trajectory distribution predictions.
APA
Shahroudi, N., Lepson, M. & Kull, M.. (2024). Evaluation of Trajectory Distribution Predictions with Energy Score. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:44322-44341 Available from https://proceedings.mlr.press/v235/shahroudi24a.html.

Related Material