Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data

Alejandro Escontrela, Justin Kerr, Kyle Stachowicz, Pieter Abbeel
Proceedings of The 8th Conference on Robot Learning, PMLR 270:5434-5445, 2025.

Abstract

Learning reliable affordance models which satisfy human preferences is often hindered by a lack of high-quality training data. Similarly, learning visuomotor policies in simulation can be challenging due to the high cost of photo-realistic rendering. We present PAWS: a comprehensive robot learning framework that uses a novel portable data capture rig and processing pipeline to collect long-horizon trajectories that include camera poses, foot poses, terrain meshes, and 3D radiance fields. We also contribute PAWS-Data: an extensive dataset gathered with PAWS containing over 10 hours of indoor and outdoor trajectories spanning a variety of scenes. With PAWS-Data we leverage radiance fields’ photo-realistic rendering to generate tens of thousands of viewpoint-augmented images, then produce pixel affordance labels by identifying semantically similar regions to those traversed by the user. On this data we finetune a navigation affordance model from a pretrained backbone, and perform detailed ablations. Additionally, We open source PAWS-Sim, a high-speed photo-realistic simulator which integrates PAWS-Data with IsaacSim, enabling research for visuomotor policy learning. We evaluate the utility of the affordance model on a quadrupedal robot, which plans through affordances to follow pathways and sidewalks, and avoid human collisions. Project resources are available on the [website](https://pawslocomotion.com).

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-escontrela25a, title = {Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data}, author = {Escontrela, Alejandro and Kerr, Justin and Stachowicz, Kyle and Abbeel, Pieter}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {5434--5445}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/escontrela25a/escontrela25a.pdf}, url = {https://proceedings.mlr.press/v270/escontrela25a.html}, abstract = {Learning reliable affordance models which satisfy human preferences is often hindered by a lack of high-quality training data. Similarly, learning visuomotor policies in simulation can be challenging due to the high cost of photo-realistic rendering. We present PAWS: a comprehensive robot learning framework that uses a novel portable data capture rig and processing pipeline to collect long-horizon trajectories that include camera poses, foot poses, terrain meshes, and 3D radiance fields. We also contribute PAWS-Data: an extensive dataset gathered with PAWS containing over 10 hours of indoor and outdoor trajectories spanning a variety of scenes. With PAWS-Data we leverage radiance fields’ photo-realistic rendering to generate tens of thousands of viewpoint-augmented images, then produce pixel affordance labels by identifying semantically similar regions to those traversed by the user. On this data we finetune a navigation affordance model from a pretrained backbone, and perform detailed ablations. Additionally, We open source PAWS-Sim, a high-speed photo-realistic simulator which integrates PAWS-Data with IsaacSim, enabling research for visuomotor policy learning. We evaluate the utility of the affordance model on a quadrupedal robot, which plans through affordances to follow pathways and sidewalks, and avoid human collisions. Project resources are available on the [website](https://pawslocomotion.com).} }
Endnote
%0 Conference Paper %T Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data %A Alejandro Escontrela %A Justin Kerr %A Kyle Stachowicz %A Pieter Abbeel %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-escontrela25a %I PMLR %P 5434--5445 %U https://proceedings.mlr.press/v270/escontrela25a.html %V 270 %X Learning reliable affordance models which satisfy human preferences is often hindered by a lack of high-quality training data. Similarly, learning visuomotor policies in simulation can be challenging due to the high cost of photo-realistic rendering. We present PAWS: a comprehensive robot learning framework that uses a novel portable data capture rig and processing pipeline to collect long-horizon trajectories that include camera poses, foot poses, terrain meshes, and 3D radiance fields. We also contribute PAWS-Data: an extensive dataset gathered with PAWS containing over 10 hours of indoor and outdoor trajectories spanning a variety of scenes. With PAWS-Data we leverage radiance fields’ photo-realistic rendering to generate tens of thousands of viewpoint-augmented images, then produce pixel affordance labels by identifying semantically similar regions to those traversed by the user. On this data we finetune a navigation affordance model from a pretrained backbone, and perform detailed ablations. Additionally, We open source PAWS-Sim, a high-speed photo-realistic simulator which integrates PAWS-Data with IsaacSim, enabling research for visuomotor policy learning. We evaluate the utility of the affordance model on a quadrupedal robot, which plans through affordances to follow pathways and sidewalks, and avoid human collisions. Project resources are available on the [website](https://pawslocomotion.com).
APA
Escontrela, A., Kerr, J., Stachowicz, K. & Abbeel, P.. (2025). Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:5434-5445 Available from https://proceedings.mlr.press/v270/escontrela25a.html.

Related Material