Temporal Multimodal Probabilistic Transformers for Safety Monitoring in Autonomous Driving Systems

Yehia Ahmed, Felippe Roza, Núria Mata
Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:535-554, 2025.

Abstract

Ensuring reliable safety monitoring in autonomous driving systems (ADS) under uncertainty is essential for deployment in real-world scenarios. We propose the Temporal Multimodal Probabilistic Transformer (TMPT), a novel deep learning framework that integrates uncertainty quantification (UQ) into lane-keeping safety monitoring. TMPT forecasts lane deviation metrics along with calibrated aleatoric and epistemic uncertainties by processing sequences of multimodal sensor and control data. Our framework combines Transformer-based temporal fusion with deep ensembles and post-hoc calibration to improve predictive accuracy and uncertainty estimation. We evaluate 24 model variants in the CARLA simulator, analyzing the impact of architecture, calibration, and ensembling on both prediction and uncertainty. Calibrated models achieve near-perfect uncertainty reliability ($ENCE < 0.03$), while uncalibrated models show sharper predictions but overconfident errors. Ensemble methods further improve robustness but incur significant computational cost. Our findings show that aligning model selection with application context–balancing precision, calibration, and efficiency–is critical for safe and practical ADS deployment.

Cite this Paper

BibTeX
@InProceedings{pmlr-v266-ahmed25a, title = {Temporal Multimodal Probabilistic Transformers for Safety Monitoring in Autonomous Driving Systems}, author = {Ahmed, Yehia and Roza, Felippe and Mata, N\'{u}ria}, booktitle = {Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications}, pages = {535--554}, year = {2025}, editor = {Nguyen, Khuong An and Luo, Zhiyuan and Papadopoulos, Harris and Löfström, Tuwe and Carlsson, Lars and Boström, Henrik}, volume = {266}, series = {Proceedings of Machine Learning Research}, month = {10--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v266/main/assets/ahmed25a/ahmed25a.pdf}, url = {https://proceedings.mlr.press/v266/ahmed25a.html}, abstract = {Ensuring reliable safety monitoring in autonomous driving systems (ADS) under uncertainty is essential for deployment in real-world scenarios. We propose the Temporal Multimodal Probabilistic Transformer (TMPT), a novel deep learning framework that integrates uncertainty quantification (UQ) into lane-keeping safety monitoring. TMPT forecasts lane deviation metrics along with calibrated aleatoric and epistemic uncertainties by processing sequences of multimodal sensor and control data. Our framework combines Transformer-based temporal fusion with deep ensembles and post-hoc calibration to improve predictive accuracy and uncertainty estimation. We evaluate 24 model variants in the CARLA simulator, analyzing the impact of architecture, calibration, and ensembling on both prediction and uncertainty. Calibrated models achieve near-perfect uncertainty reliability ($ENCE < 0.03$), while uncalibrated models show sharper predictions but overconfident errors. Ensemble methods further improve robustness but incur significant computational cost. Our findings show that aligning model selection with application context–balancing precision, calibration, and efficiency–is critical for safe and practical ADS deployment.} }
Endnote
%0 Conference Paper %T Temporal Multimodal Probabilistic Transformers for Safety Monitoring in Autonomous Driving Systems %A Yehia Ahmed %A Felippe Roza %A Núria Mata %B Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications %C Proceedings of Machine Learning Research %D 2025 %E Khuong An Nguyen %E Zhiyuan Luo %E Harris Papadopoulos %E Tuwe Löfström %E Lars Carlsson %E Henrik Boström %F pmlr-v266-ahmed25a %I PMLR %P 535--554 %U https://proceedings.mlr.press/v266/ahmed25a.html %V 266 %X Ensuring reliable safety monitoring in autonomous driving systems (ADS) under uncertainty is essential for deployment in real-world scenarios. We propose the Temporal Multimodal Probabilistic Transformer (TMPT), a novel deep learning framework that integrates uncertainty quantification (UQ) into lane-keeping safety monitoring. TMPT forecasts lane deviation metrics along with calibrated aleatoric and epistemic uncertainties by processing sequences of multimodal sensor and control data. Our framework combines Transformer-based temporal fusion with deep ensembles and post-hoc calibration to improve predictive accuracy and uncertainty estimation. We evaluate 24 model variants in the CARLA simulator, analyzing the impact of architecture, calibration, and ensembling on both prediction and uncertainty. Calibrated models achieve near-perfect uncertainty reliability ($ENCE < 0.03$), while uncalibrated models show sharper predictions but overconfident errors. Ensemble methods further improve robustness but incur significant computational cost. Our findings show that aligning model selection with application context–balancing precision, calibration, and efficiency–is critical for safe and practical ADS deployment.
APA
Ahmed, Y., Roza, F. & Mata, N.. (2025). Temporal Multimodal Probabilistic Transformers for Safety Monitoring in Autonomous Driving Systems. Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, in Proceedings of Machine Learning Research 266:535-554 Available from https://proceedings.mlr.press/v266/ahmed25a.html.

Related Material