Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting

Parth Kothari, Danya Li, Yuejiang Liu, Alexandre Alahi
Proceedings of The 6th Conference on Robot Learning, PMLR 205:774-784, 2023.

Abstract

Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-kothari23a, title = {Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting}, author = {Kothari, Parth and Li, Danya and Liu, Yuejiang and Alahi, Alexandre}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {774--784}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/kothari23a/kothari23a.pdf}, url = {https://proceedings.mlr.press/v205/kothari23a.html}, abstract = {Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.} }
Endnote
%0 Conference Paper %T Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting %A Parth Kothari %A Danya Li %A Yuejiang Liu %A Alexandre Alahi %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-kothari23a %I PMLR %P 774--784 %U https://proceedings.mlr.press/v205/kothari23a.html %V 205 %X Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.
APA
Kothari, P., Li, D., Liu, Y. & Alahi, A.. (2023). Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:774-784 Available from https://proceedings.mlr.press/v205/kothari23a.html.

Related Material