Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

Sam Devlin, Raluca Georgescu, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, Katja Hofmann
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:2644-2653, 2021.

Abstract

A key challenge on the path to developing agents that learn complex human-like behavior is the need to quickly and accurately quantify human-likeness. While human assessments of such behavior can be highly accurate, speed and scalability are limited. We address these limitations through a novel automated Navigation Turing Test (ANTT) that learns to predict human judgments of human-likeness. We demonstrate the effectiveness of our automated NTT on a navigation task in a complex 3D environment. We investigate six classification models to shed light on the types of architectures best suited to this task, and validate them against data collected through a human NTT. Our best models achieve high accuracy when distinguishing true human and agent behavior. At the same time, we show that predicting finer-grained human assessment of agents’ progress towards human-like behavior remains unsolved. Our work takes an important step towards agents that more effectively learn complex human-like behavior.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-devlin21a, title = {Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation}, author = {Devlin, Sam and Georgescu, Raluca and Momennejad, Ida and Rzepecki, Jaroslaw and Zuniga, Evelyn and Costello, Gavin and Leroy, Guy and Shaw, Ali and Hofmann, Katja}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {2644--2653}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/devlin21a/devlin21a.pdf}, url = {https://proceedings.mlr.press/v139/devlin21a.html}, abstract = {A key challenge on the path to developing agents that learn complex human-like behavior is the need to quickly and accurately quantify human-likeness. While human assessments of such behavior can be highly accurate, speed and scalability are limited. We address these limitations through a novel automated Navigation Turing Test (ANTT) that learns to predict human judgments of human-likeness. We demonstrate the effectiveness of our automated NTT on a navigation task in a complex 3D environment. We investigate six classification models to shed light on the types of architectures best suited to this task, and validate them against data collected through a human NTT. Our best models achieve high accuracy when distinguishing true human and agent behavior. At the same time, we show that predicting finer-grained human assessment of agents’ progress towards human-like behavior remains unsolved. Our work takes an important step towards agents that more effectively learn complex human-like behavior.} }
Endnote
%0 Conference Paper %T Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation %A Sam Devlin %A Raluca Georgescu %A Ida Momennejad %A Jaroslaw Rzepecki %A Evelyn Zuniga %A Gavin Costello %A Guy Leroy %A Ali Shaw %A Katja Hofmann %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-devlin21a %I PMLR %P 2644--2653 %U https://proceedings.mlr.press/v139/devlin21a.html %V 139 %X A key challenge on the path to developing agents that learn complex human-like behavior is the need to quickly and accurately quantify human-likeness. While human assessments of such behavior can be highly accurate, speed and scalability are limited. We address these limitations through a novel automated Navigation Turing Test (ANTT) that learns to predict human judgments of human-likeness. We demonstrate the effectiveness of our automated NTT on a navigation task in a complex 3D environment. We investigate six classification models to shed light on the types of architectures best suited to this task, and validate them against data collected through a human NTT. Our best models achieve high accuracy when distinguishing true human and agent behavior. At the same time, we show that predicting finer-grained human assessment of agents’ progress towards human-like behavior remains unsolved. Our work takes an important step towards agents that more effectively learn complex human-like behavior.
APA
Devlin, S., Georgescu, R., Momennejad, I., Rzepecki, J., Zuniga, E., Costello, G., Leroy, G., Shaw, A. & Hofmann, K.. (2021). Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:2644-2653 Available from https://proceedings.mlr.press/v139/devlin21a.html.

Related Material