Velociraptor: Leveraging Visual Foundation Models for Label-Free, Risk-Aware Off-Road Navigation

Samuel Triest, Matthew Sivaprakasam, Shubhra Aich, David Fan, Wenshan Wang, Sebastian Scherer
Proceedings of The 8th Conference on Robot Learning, PMLR 270:4483-4494, 2025.

Abstract

Traversability analysis in off-road regimes is a challenging task that requires understanding of multi-modal inputs such as camera and LiDAR. These measurements are often sparse, noisy, and difficult to interpret, particularly in the off-road setting. Existing systems are very engineering-intensive, often requiring hand-tuning of traversability rules and manual annotation of semantic labels. Furthermore, existing methods for analyzing traversability risk and uncertainty are computationally expensive or not well-calibrated. We propose Velociraptor, a traversability analysis system that performs [veloci]ty-informed, [r]isk-[a]ware [p]erception and [t]raversability for [o]ff-[r]oad driving without any human annotations. We achieve this via the use of visual foundation models (VFMs) and geometric mapping to produce a rich visual-geometric representation of the robot’s local environment. We then leverage this representation to produce costmaps, speedmaps, and uncertainty maps using state-of-the-art fully self-supervised techniques. Our approach enables intelligent high-speed off-road navigation with zero human annotation, and with about forty minutes of expert data, outperforms several geometric and semantic traversability baselines, both in offline and real-world robot trials across multiple challenging off-road sites.

Cite this Paper

BibTeX
@InProceedings{pmlr-v270-triest25a, title = {Velociraptor: Leveraging Visual Foundation Models for Label-Free, Risk-Aware Off-Road Navigation}, author = {Triest, Samuel and Sivaprakasam, Matthew and Aich, Shubhra and Fan, David and Wang, Wenshan and Scherer, Sebastian}, booktitle = {Proceedings of The 8th Conference on Robot Learning}, pages = {4483--4494}, year = {2025}, editor = {Agrawal, Pulkit and Kroemer, Oliver and Burgard, Wolfram}, volume = {270}, series = {Proceedings of Machine Learning Research}, month = {06--09 Nov}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v270/main/assets/triest25a/triest25a.pdf}, url = {https://proceedings.mlr.press/v270/triest25a.html}, abstract = {Traversability analysis in off-road regimes is a challenging task that requires understanding of multi-modal inputs such as camera and LiDAR. These measurements are often sparse, noisy, and difficult to interpret, particularly in the off-road setting. Existing systems are very engineering-intensive, often requiring hand-tuning of traversability rules and manual annotation of semantic labels. Furthermore, existing methods for analyzing traversability risk and uncertainty are computationally expensive or not well-calibrated. We propose Velociraptor, a traversability analysis system that performs [veloci]ty-informed, [r]isk-[a]ware [p]erception and [t]raversability for [o]ff-[r]oad driving without any human annotations. We achieve this via the use of visual foundation models (VFMs) and geometric mapping to produce a rich visual-geometric representation of the robot’s local environment. We then leverage this representation to produce costmaps, speedmaps, and uncertainty maps using state-of-the-art fully self-supervised techniques. Our approach enables intelligent high-speed off-road navigation with zero human annotation, and with about forty minutes of expert data, outperforms several geometric and semantic traversability baselines, both in offline and real-world robot trials across multiple challenging off-road sites.} }
Endnote
%0 Conference Paper %T Velociraptor: Leveraging Visual Foundation Models for Label-Free, Risk-Aware Off-Road Navigation %A Samuel Triest %A Matthew Sivaprakasam %A Shubhra Aich %A David Fan %A Wenshan Wang %A Sebastian Scherer %B Proceedings of The 8th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2025 %E Pulkit Agrawal %E Oliver Kroemer %E Wolfram Burgard %F pmlr-v270-triest25a %I PMLR %P 4483--4494 %U https://proceedings.mlr.press/v270/triest25a.html %V 270 %X Traversability analysis in off-road regimes is a challenging task that requires understanding of multi-modal inputs such as camera and LiDAR. These measurements are often sparse, noisy, and difficult to interpret, particularly in the off-road setting. Existing systems are very engineering-intensive, often requiring hand-tuning of traversability rules and manual annotation of semantic labels. Furthermore, existing methods for analyzing traversability risk and uncertainty are computationally expensive or not well-calibrated. We propose Velociraptor, a traversability analysis system that performs [veloci]ty-informed, [r]isk-[a]ware [p]erception and [t]raversability for [o]ff-[r]oad driving without any human annotations. We achieve this via the use of visual foundation models (VFMs) and geometric mapping to produce a rich visual-geometric representation of the robot’s local environment. We then leverage this representation to produce costmaps, speedmaps, and uncertainty maps using state-of-the-art fully self-supervised techniques. Our approach enables intelligent high-speed off-road navigation with zero human annotation, and with about forty minutes of expert data, outperforms several geometric and semantic traversability baselines, both in offline and real-world robot trials across multiple challenging off-road sites.
APA
Triest, S., Sivaprakasam, M., Aich, S., Fan, D., Wang, W. & Scherer, S.. (2025). Velociraptor: Leveraging Visual Foundation Models for Label-Free, Risk-Aware Off-Road Navigation. Proceedings of The 8th Conference on Robot Learning, in Proceedings of Machine Learning Research 270:4483-4494 Available from https://proceedings.mlr.press/v270/triest25a.html.

Related Material