Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation

Rachel Luo, Heng Yang, Michael Watson, Apoorva Sharma, Sushant Veer, Edward Schmerling, Marco Pavone
Proceedings of The 9th Conference on Robot Learning, PMLR 305:2294-2310, 2025.

Abstract

Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive and if conducted may still yield insufficient data for high-confidence guarantees. In this work, we introduce a general estimation framework that leverages *paired* data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly reduced sample complexity. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v305-luo25a, title = {Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation}, author = {Luo, Rachel and Yang, Heng and Watson, Michael and Sharma, Apoorva and Veer, Sushant and Schmerling, Edward and Pavone, Marco}, booktitle = {Proceedings of The 9th Conference on Robot Learning}, pages = {2294--2310}, year = {2025}, editor = {Lim, Joseph and Song, Shuran and Park, Hae-Won}, volume = {305}, series = {Proceedings of Machine Learning Research}, month = {27--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v305/main/assets/luo25a/luo25a.pdf}, url = {https://proceedings.mlr.press/v305/luo25a.html}, abstract = {Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive and if conducted may still yield insufficient data for high-confidence guarantees. In this work, we introduce a general estimation framework that leverages *paired* data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly reduced sample complexity. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems.} }
Endnote
%0 Conference Paper %T Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation %A Rachel Luo %A Heng Yang %A Michael Watson %A Apoorva Sharma %A Sushant Veer %A Edward Schmerling %A Marco Pavone %B Proceedings of The 9th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Joseph Lim %E Shuran Song %E Hae-Won Park %F pmlr-v305-luo25a %I PMLR %P 2294--2310 %U https://proceedings.mlr.press/v305/luo25a.html %V 305 %X Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive and if conducted may still yield insufficient data for high-confidence guarantees. In this work, we introduce a general estimation framework that leverages *paired* data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly reduced sample complexity. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems.
APA
Luo, R., Yang, H., Watson, M., Sharma, A., Veer, S., Schmerling, E. & Pavone, M.. (2025). Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation. Proceedings of The 9th Conference on Robot Learning, in Proceedings of Machine Learning Research 305:2294-2310 Available from https://proceedings.mlr.press/v305/luo25a.html.

Related Material