Learning to be Multimodal : Co-evolving Sensory Modalities and Sensor Properties

Rika Antonova, Jeannette Bohg
Proceedings of the 5th Conference on Robot Learning, PMLR 164:1782-1788, 2022.

Abstract

Making a single sensory modality precise and robust enough to get human-level performance and autonomy could be very expensive or intractable. Fusing information from multiple sensory modalities is promising – for example, recent works showed benefits from combining vision with haptic sensors or with audio data. Learning-based methods facilitate faster progress in this field by removing the need for manual feature engineering. However, the sensor properties and the choice of sensory modalities is still usually done manually. Our blue-sky view is that we could simulate/emulate sensors with various properties, then infer which properties and combinations of sensors yield the best learning outcomes. This view would incentivize the development of novel, affordable sensors that can make a noticeable impact on the performance, robustness and ease of training classifiers, models and policies for robotics. This would motivate making hardware that provides signals complementary to the existing ones. As a result: we can significantly expand the realm of applicability of the learning-based approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v164-antonova22a, title = {Learning to be Multimodal : Co-evolving Sensory Modalities and Sensor Properties}, author = {Antonova, Rika and Bohg, Jeannette}, booktitle = {Proceedings of the 5th Conference on Robot Learning}, pages = {1782--1788}, year = {2022}, editor = {Faust, Aleksandra and Hsu, David and Neumann, Gerhard}, volume = {164}, series = {Proceedings of Machine Learning Research}, month = {08--11 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v164/antonova22a/antonova22a.pdf}, url = {https://proceedings.mlr.press/v164/antonova22a.html}, abstract = {Making a single sensory modality precise and robust enough to get human-level performance and autonomy could be very expensive or intractable. Fusing information from multiple sensory modalities is promising – for example, recent works showed benefits from combining vision with haptic sensors or with audio data. Learning-based methods facilitate faster progress in this field by removing the need for manual feature engineering. However, the sensor properties and the choice of sensory modalities is still usually done manually. Our blue-sky view is that we could simulate/emulate sensors with various properties, then infer which properties and combinations of sensors yield the best learning outcomes. This view would incentivize the development of novel, affordable sensors that can make a noticeable impact on the performance, robustness and ease of training classifiers, models and policies for robotics. This would motivate making hardware that provides signals complementary to the existing ones. As a result: we can significantly expand the realm of applicability of the learning-based approaches.} }
Endnote
%0 Conference Paper %T Learning to be Multimodal : Co-evolving Sensory Modalities and Sensor Properties %A Rika Antonova %A Jeannette Bohg %B Proceedings of the 5th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2022 %E Aleksandra Faust %E David Hsu %E Gerhard Neumann %F pmlr-v164-antonova22a %I PMLR %P 1782--1788 %U https://proceedings.mlr.press/v164/antonova22a.html %V 164 %X Making a single sensory modality precise and robust enough to get human-level performance and autonomy could be very expensive or intractable. Fusing information from multiple sensory modalities is promising – for example, recent works showed benefits from combining vision with haptic sensors or with audio data. Learning-based methods facilitate faster progress in this field by removing the need for manual feature engineering. However, the sensor properties and the choice of sensory modalities is still usually done manually. Our blue-sky view is that we could simulate/emulate sensors with various properties, then infer which properties and combinations of sensors yield the best learning outcomes. This view would incentivize the development of novel, affordable sensors that can make a noticeable impact on the performance, robustness and ease of training classifiers, models and policies for robotics. This would motivate making hardware that provides signals complementary to the existing ones. As a result: we can significantly expand the realm of applicability of the learning-based approaches.
APA
Antonova, R. & Bohg, J.. (2022). Learning to be Multimodal : Co-evolving Sensory Modalities and Sensor Properties. Proceedings of the 5th Conference on Robot Learning, in Proceedings of Machine Learning Research 164:1782-1788 Available from https://proceedings.mlr.press/v164/antonova22a.html.

Related Material