Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning

Dhruva Tirumala, Markus Wulfmeier, Ben Moran, Sandy Huang, Jan Humplik, Guy Lever, Tuomas Haarnoja, Leonard Hasenclever, Arunkumar Byravan, Nathan Batchelor, Neil sreendra, Kushal Patel, Marlon Gwira, Francesco Nori, Martin Riedmiller, Nicolas Heess
Proceedings of The 8th Conference on Robot Learning, PMLR 270:165-184, 2025.

Abstract

We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting reflects many challenges of real-world robotics, including active perception, agile full-body control, and long-horizon planning in a dynamic, partially-observable, multi-agent domain. We rely on large-scale, simulation-based data generation to obtain complex behaviors from egocentric vision which can be successfully transferred to physical robots using low-cost sensors. To achieve adequate visual realism, our simulation combines rigid-body physics with learned, realistic rendering via multiple Neural Radiance Fields (NeRFs). We combine teacher-based multi-agent RL and cross-experiment data reuse to enable the discovery of sophisticated soccer strategies. We analyze active-perception behaviors including object tracking and ball seeking that emerge when simply optimizing perception-agnostic soccer play. The agents display equivalent levels of performance and agility as policies with access to privileged, ground-truth state. To our knowledge, this paper constitutes a first demonstration of end-to-end training for multi-agent robot soccer, mapping raw pixel observations to joint-level actions that can be deployed in the real world.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-tirumala25a, title = {Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning}, author = {Tirumala, Dhruva and Wulfmeier, Markus and Moran, Ben and Huang, Sandy and Humplik, Jan and Lever, Guy and Haarnoja, Tuomas and Hasenclever, Leonard and Byravan, Arunkumar and Batchelor, Nathan and sreendra, Neil and Patel, Kushal and Gwira, Marlon and Nori, Francesco and Riedmiller, Martin and Heess, Nicolas}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {165--184}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/tirumala25a/tirumala25a.pdf}, url = {https://proceedings.mlr.press/v270/tirumala25a.html}, abstract = {We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting reflects many challenges of real-world robotics, including active perception, agile full-body control, and long-horizon planning in a dynamic, partially-observable, multi-agent domain. We rely on large-scale, simulation-based data generation to obtain complex behaviors from egocentric vision which can be successfully transferred to physical robots using low-cost sensors. To achieve adequate visual realism, our simulation combines rigid-body physics with learned, realistic rendering via multiple Neural Radiance Fields (NeRFs). We combine teacher-based multi-agent RL and cross-experiment data reuse to enable the discovery of sophisticated soccer strategies. We analyze active-perception behaviors including object tracking and ball seeking that emerge when simply optimizing perception-agnostic soccer play. The agents display equivalent levels of performance and agility as policies with access to privileged, ground-truth state. To our knowledge, this paper constitutes a first demonstration of end-to-end training for multi-agent robot soccer, mapping raw pixel observations to joint-level actions that can be deployed in the real world.} }
Endnote
%0 Conference Paper %T Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning %A Dhruva Tirumala %A Markus Wulfmeier %A Ben Moran %A Sandy Huang %A Jan Humplik %A Guy Lever %A Tuomas Haarnoja %A Leonard Hasenclever %A Arunkumar Byravan %A Nathan Batchelor %A Neil sreendra %A Kushal Patel %A Marlon Gwira %A Francesco Nori %A Martin Riedmiller %A Nicolas Heess %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-tirumala25a %I PMLR %P 165--184 %U https://proceedings.mlr.press/v270/tirumala25a.html %V 270 %X We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting reflects many challenges of real-world robotics, including active perception, agile full-body control, and long-horizon planning in a dynamic, partially-observable, multi-agent domain. We rely on large-scale, simulation-based data generation to obtain complex behaviors from egocentric vision which can be successfully transferred to physical robots using low-cost sensors. To achieve adequate visual realism, our simulation combines rigid-body physics with learned, realistic rendering via multiple Neural Radiance Fields (NeRFs). We combine teacher-based multi-agent RL and cross-experiment data reuse to enable the discovery of sophisticated soccer strategies. We analyze active-perception behaviors including object tracking and ball seeking that emerge when simply optimizing perception-agnostic soccer play. The agents display equivalent levels of performance and agility as policies with access to privileged, ground-truth state. To our knowledge, this paper constitutes a first demonstration of end-to-end training for multi-agent robot soccer, mapping raw pixel observations to joint-level actions that can be deployed in the real world.
APA
Tirumala, D., Wulfmeier, M., Moran, B., Huang, S., Humplik, J., Lever, G., Haarnoja, T., Hasenclever, L., Byravan, A., Batchelor, N., sreendra, N., Patel, K., Gwira, M., Nori, F., Riedmiller, M. & Heess, N.. (2025). Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:165-184 Available from https://proceedings.mlr.press/v270/tirumala25a.html.

Related Material