UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

Hanjung Kim, Jaehyun Kang, Hyolim Kang, Meedeum Cho, Seon Joo Kim, Youngwoon Lee
Proceedings of The 9th Conference on Robot Learning, PMLR 305:4269-4294, 2025.

Abstract

Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-kim25d, title = {UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations}, author = {Kim, Hanjung and Kang, Jaehyun and Kang, Hyolim and Cho, Meedeum and Kim, Seon Joo and Lee, Youngwoon}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {4269--4294}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/kim25d/kim25d.pdf}, url = {https://proceedings.mlr.press/v305/kim25d.html}, abstract = {Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts.} }
Endnote
%0 Conference Paper %T UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations %A Hanjung Kim %A Jaehyun Kang %A Hyolim Kang %A Meedeum Cho %A Seon Joo Kim %A Youngwoon Lee %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-kim25d %I PMLR %P 4269--4294 %U https://proceedings.mlr.press/v305/kim25d.html %V 305 %X Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts.
APA
Kim, H., Kang, J., Kang, H., Cho, M., Kim, S.J. & Lee, Y.. (2025). UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:4269-4294 Available from https://proceedings.mlr.press/v305/kim25d.html.

Related Material