Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments

Martin Weiss, Simon Chamorro, Roger Girgis, Margaux Luck, Samira E. Kahou, Joseph P. Cohen, Derek Nowrouzezahrai, Doina Precup, Florian Golemo, Chris Pal
Proceedings of the Conference on Robot Learning, PMLR 100:1314-1327, 2020.

Abstract

Millions of blind and visually-impaired (BVI) people navigate urban environments everyday, using smartphones for high-level path-planning and white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavour to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v100-weiss20a, title = {Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments}, author = {Weiss, Martin and Chamorro, Simon and Girgis, Roger and Luck, Margaux and Kahou, Samira E. and Cohen, Joseph P. and Nowrouzezahrai, Derek and Precup, Doina and Golemo, Florian and Pal, Chris}, booktitle = {Proceedings of the Conference on Robot Learning}, pages = {1314--1327}, year = {2020}, editor = {Kaelbling, Leslie Pack and Kragic, Danica and Sugiura, Komei}, volume = {100}, series = {Proceedings of Machine Learning Research}, month = {30 Oct--01 Nov}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v100/weiss20a/weiss20a.pdf}, url = {https://proceedings.mlr.press/v100/weiss20a.html}, abstract = {Millions of blind and visually-impaired (BVI) people navigate urban environments everyday, using smartphones for high-level path-planning and white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavour to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.} }
Endnote
%0 Conference Paper %T Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments %A Martin Weiss %A Simon Chamorro %A Roger Girgis %A Margaux Luck %A Samira E. Kahou %A Joseph P. Cohen %A Derek Nowrouzezahrai %A Doina Precup %A Florian Golemo %A Chris Pal %B Proceedings of the Conference on Robot Learning %C Proceedings of Machine Learning Research %D 2020 %E Leslie Pack Kaelbling %E Danica Kragic %E Komei Sugiura %F pmlr-v100-weiss20a %I PMLR %P 1314--1327 %U https://proceedings.mlr.press/v100/weiss20a.html %V 100 %X Millions of blind and visually-impaired (BVI) people navigate urban environments everyday, using smartphones for high-level path-planning and white canes or guide dogs for local information. However, many BVI people still struggle to travel to new places. In our endeavour to create a navigation assistant for the BVI, we found that existing Reinforcement Learning (RL) environments were unsuitable for the task. This work introduces SEVN, a sidewalk simulation environment and a neural network-based approach to creating a navigation agent. SEVN contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. We study the performance of an RL algorithm (PPO) in this setting. Our policy model fuses multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data to navigate to a goal door. We hope that this dataset, simulator, and experimental results will provide a foundation for further research into the creation of agents that can assist members of the BVI community with outdoor navigation.
APA
Weiss, M., Chamorro, S., Girgis, R., Luck, M., Kahou, S.E., Cohen, J.P., Nowrouzezahrai, D., Precup, D., Golemo, F. & Pal, C.. (2020). Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments. Proceedings of the Conference on Robot Learning, in Proceedings of Machine Learning Research 100:1314-1327 Available from https://proceedings.mlr.press/v100/weiss20a.html.

Related Material