Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation

Xuesu Xiao, Tingnan Zhang, Krzysztof Marcin Choromanski, Tsang-Wei Edward Lee, Anthony Francis, Jake Varley, Stephen Tu, Sumeet Singh, Peng Xu, Fei Xia, Sven Mikael Persson, Dmitry Kalashnikov, Leila Takayama, Roy Frostig, Jie Tan, Carolina Parada, Vikas Sindhwani
Proceedings of The 6th Conference on Robot Learning, PMLR 205:1708-1721, 2023.

Abstract

Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied public spaces. To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the robust handling of system constraints from Model Predictive Control (MPC). Our approach, called Performer-MPC, uses a learned cost function parameterized by vision context embeddings provided by Performers—a low-rank implicit-attention Transformer. We jointly train the cost function and construct the controller relying on it, effectively solving end-to-end the corresponding bi-level optimization problem. We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in different challenging real-world scenarios. Compared with a standard MPC policy, Performer-MPC achieves >40% better goal reached in cluttered environments and >65% better on social metrics when navigating around humans.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-xiao23a, title = {Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation}, author = {Xiao, Xuesu and Zhang, Tingnan and Choromanski, Krzysztof Marcin and Lee, Tsang-Wei Edward and Francis, Anthony and Varley, Jake and Tu, Stephen and Singh, Sumeet and Xu, Peng and Xia, Fei and Persson, Sven Mikael and Kalashnikov, Dmitry and Takayama, Leila and Frostig, Roy and Tan, Jie and Parada, Carolina and Sindhwani, Vikas}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {1708--1721}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/xiao23a/xiao23a.pdf}, url = {https://proceedings.mlr.press/v205/xiao23a.html}, abstract = {Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied public spaces. To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the robust handling of system constraints from Model Predictive Control (MPC). Our approach, called Performer-MPC, uses a learned cost function parameterized by vision context embeddings provided by Performers—a low-rank implicit-attention Transformer. We jointly train the cost function and construct the controller relying on it, effectively solving end-to-end the corresponding bi-level optimization problem. We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in different challenging real-world scenarios. Compared with a standard MPC policy, Performer-MPC achieves >40% better goal reached in cluttered environments and >65% better on social metrics when navigating around humans. } }
Endnote
%0 Conference Paper %T Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation %A Xuesu Xiao %A Tingnan Zhang %A Krzysztof Marcin Choromanski %A Tsang-Wei Edward Lee %A Anthony Francis %A Jake Varley %A Stephen Tu %A Sumeet Singh %A Peng Xu %A Fei Xia %A Sven Mikael Persson %A Dmitry Kalashnikov %A Leila Takayama %A Roy Frostig %A Jie Tan %A Carolina Parada %A Vikas Sindhwani %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-xiao23a %I PMLR %P 1708--1721 %U https://proceedings.mlr.press/v205/xiao23a.html %V 205 %X Despite decades of research, existing navigation systems still face real-world challenges when deployed in the wild, e.g., in cluttered home environments or in human-occupied public spaces. To address this, we present a new class of implicit control policies combining the benefits of imitation learning with the robust handling of system constraints from Model Predictive Control (MPC). Our approach, called Performer-MPC, uses a learned cost function parameterized by vision context embeddings provided by Performers—a low-rank implicit-attention Transformer. We jointly train the cost function and construct the controller relying on it, effectively solving end-to-end the corresponding bi-level optimization problem. We show that the resulting policy improves standard MPC performance by leveraging a few expert demonstrations of the desired navigation behavior in different challenging real-world scenarios. Compared with a standard MPC policy, Performer-MPC achieves >40% better goal reached in cluttered environments and >65% better on social metrics when navigating around humans.
APA
Xiao, X., Zhang, T., Choromanski, K.M., Lee, T.E., Francis, A., Varley, J., Tu, S., Singh, S., Xu, P., Xia, F., Persson, S.M., Kalashnikov, D., Takayama, L., Frostig, R., Tan, J., Parada, C. & Sindhwani, V.. (2023). Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:1708-1721 Available from https://proceedings.mlr.press/v205/xiao23a.html.

Related Material