Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot

Debasmita Ghose, Michal Adam Lewkowicz, Kaleb Gezahegn, Julian Lee, Timothy Adamson, Marynel Vazquez, Brian Scassellati
Proceedings of The 6th Conference on Robot Learning, PMLR 205:583-593, 2023.

Abstract

Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks. In many applications though, a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre-program visual object classifiers suitable for the task of interest. In this work, we study the problem of learning a classifier for visual objects based on a few examples provided by humans. We frame this problem from the perspective of learning a suitable visual object representation that allows us to distinguish the desired object category from others. Our proposed approach integrates human supervision into the representation learning process by combining contrastive learning with an additional loss function that brings the representations of human examples close to each other in the latent space. Our experiments show that our proposed method performs better than self-supervised and fully supervised learning methods in offline evaluations and can also be used in real-time by a robot in a simplified recycling domain, where recycling streams contain a variety of objects.

Cite this Paper

BibTeX
@InProceedings{pmlr-v205-ghose23a, title = {Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot}, author = {Ghose, Debasmita and Lewkowicz, Michal Adam and Gezahegn, Kaleb and Lee, Julian and Adamson, Timothy and Vazquez, Marynel and Scassellati, Brian}, booktitle = {Proceedings of The 6th Conference on Robot Learning}, pages = {583--593}, year = {2023}, editor = {Liu, Karen and Kulic, Dana and Ichnowski, Jeff}, volume = {205}, series = {Proceedings of Machine Learning Research}, month = {14--18 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v205/ghose23a/ghose23a.pdf}, url = {https://proceedings.mlr.press/v205/ghose23a.html}, abstract = {Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks. In many applications though, a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre-program visual object classifiers suitable for the task of interest. In this work, we study the problem of learning a classifier for visual objects based on a few examples provided by humans. We frame this problem from the perspective of learning a suitable visual object representation that allows us to distinguish the desired object category from others. Our proposed approach integrates human supervision into the representation learning process by combining contrastive learning with an additional loss function that brings the representations of human examples close to each other in the latent space. Our experiments show that our proposed method performs better than self-supervised and fully supervised learning methods in offline evaluations and can also be used in real-time by a robot in a simplified recycling domain, where recycling streams contain a variety of objects.} }
Endnote
%0 Conference Paper %T Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot %A Debasmita Ghose %A Michal Adam Lewkowicz %A Kaleb Gezahegn %A Julian Lee %A Timothy Adamson %A Marynel Vazquez %A Brian Scassellati %B Proceedings of The 6th Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2023 %E Karen Liu %E Dana Kulic %E Jeff Ichnowski %F pmlr-v205-ghose23a %I PMLR %P 583--593 %U https://proceedings.mlr.press/v205/ghose23a.html %V 205 %X Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks. In many applications though, a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre-program visual object classifiers suitable for the task of interest. In this work, we study the problem of learning a classifier for visual objects based on a few examples provided by humans. We frame this problem from the perspective of learning a suitable visual object representation that allows us to distinguish the desired object category from others. Our proposed approach integrates human supervision into the representation learning process by combining contrastive learning with an additional loss function that brings the representations of human examples close to each other in the latent space. Our experiments show that our proposed method performs better than self-supervised and fully supervised learning methods in offline evaluations and can also be used in real-time by a robot in a simplified recycling domain, where recycling streams contain a variety of objects.
APA
Ghose, D., Lewkowicz, M.A., Gezahegn, K., Lee, J., Adamson, T., Vazquez, M. & Scassellati, B.. (2023). Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot. Proceedings of The 6th Conference on Robot Learning, in Proceedings of Machine Learning Research 205:583-593 Available from https://proceedings.mlr.press/v205/ghose23a.html.

Related Material