Promptable Closed-loop Traffic Simulation

Shuhan Tan, Boris Ivanovic, Yuxiao Chen, Boyi Li, Xinshuo Weng, Yulong Cao, Philipp Kraehenbuehl, Marco Pavone
Proceedings of The 8th Conference on Robot Learning, PMLR 270:5087-5105, 2025.

Abstract

Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to produce realistic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim allows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent’s behavior and intention. ProSim then rolls out a traffic scenario in a closed-loop manner, modeling each agent’s interaction with other traffic participants. Our experiments show that ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. We will release data, benchmark, and labeling tools of ProSim-Instruct-520k upon publication.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-tan25a, title = {Promptable Closed-loop Traffic Simulation}, author = {Tan, Shuhan and Ivanovic, Boris and Chen, Yuxiao and Li, Boyi and Weng, Xinshuo and Cao, Yulong and Kraehenbuehl, Philipp and Pavone, Marco}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {5087--5105}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/tan25a/tan25a.pdf}, url = {https://proceedings.mlr.press/v270/tan25a.html}, abstract = {Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to produce realistic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim allows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent’s behavior and intention. ProSim then rolls out a traffic scenario in a closed-loop manner, modeling each agent’s interaction with other traffic participants. Our experiments show that ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. We will release data, benchmark, and labeling tools of ProSim-Instruct-520k upon publication.} }
Endnote
%0 Conference Paper %T Promptable Closed-loop Traffic Simulation %A Shuhan Tan %A Boris Ivanovic %A Yuxiao Chen %A Boyi Li %A Xinshuo Weng %A Yulong Cao %A Philipp Kraehenbuehl %A Marco Pavone %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-tan25a %I PMLR %P 5087--5105 %U https://proceedings.mlr.press/v270/tan25a.html %V 270 %X Simulation stands as a cornerstone for safe and efficient autonomous driving development. At its core a simulation system ought to produce realistic, reactive, and controllable traffic patterns. In this paper, we propose ProSim, a multimodal promptable closed-loop traffic simulation framework. ProSim allows the user to give a complex set of numerical, categorical or textual prompts to instruct each agent’s behavior and intention. ProSim then rolls out a traffic scenario in a closed-loop manner, modeling each agent’s interaction with other traffic participants. Our experiments show that ProSim achieves high prompt controllability given different user prompts, while reaching competitive performance on the Waymo Sim Agents Challenge when no prompt is given. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset with over 10M text prompts for over 520k real-world driving scenarios. We will release data, benchmark, and labeling tools of ProSim-Instruct-520k upon publication.
APA
Tan, S., Ivanovic, B., Chen, Y., Li, B., Weng, X., Cao, Y., Kraehenbuehl, P. & Pavone, M.. (2025). Promptable Closed-loop Traffic Simulation. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:5087-5105 Available from https://proceedings.mlr.press/v270/tan25a.html.

Related Material