poster

Open access

How Humans Perceive Human-like Behavior in Video Game Navigation

Authors:

Stephanie Milani,

Jaroslaw Rzepecki,

Raluca Georgescu,

Ida Momennejad,

Gavin Costello,

Katja HofmannAuthors Info & Claims

CHI EA '22: CHI Conference on Human Factors in Computing Systems Extended Abstracts

Article No.: 391, Pages 1 - 11

https://doi.org/10.1145/3491101.3519735

Published: 28 April 2022 Publication History

All formats PDF

Abstract

The goal of this paper is to understand how people assess human-likeness in human- and AI-generated behavior. To this end, we present a qualitative study of hundreds of crowd-sourced assessments of human-likeness of behavior in a 3D video game navigation task. In particular, we focus on an AI agent that has passed a Turing Test, in the sense that human judges were not able to reliably distinguish between videos of a human and AI agent navigating on a quantitative level. Our insights shine a light on the characteristics that people consider as human-like. Understanding these characteristics is a key first step for improving AI agents in the future.

Supplementary Material

MP4 File (3491101.3519735-video-preview.mp4)

Video Preview

Download
8.91 MB

MP4 File (3491101.3519735-talk-video.mp4)

Talk Video

Download
18.85 MB

References

[1]

Eloi Alonso, Ubisoft La Forge, Maxim Peter, David Goumard, and Joshua Romoff. 2020. Deep Reinforcement Learning for Navigation in AAA Video Games. In Challenges of Real-World Reinforcement Learning NeurIPS Workshop. NeurIPS, Montreal, Canada, 1–13.

[2]

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 1 (2019), 1–66.

[3]

Stephen Borthwick and Hugh Durrant-Whyte. 1994. Dynamic localisation of autonomous guided vehicles. In Proceedings of 1994 IEEE International Conference on MFI’94. Multisensor Fusion and Integration for Intelligent Systems. IEEE, Piscataway, NJ, 92–97.

[4]

Rodney A Brooks, Cynthia Breazeal, Robert Irie, Charles C Kemp, Matthew Marjanovic, Brian Scassellati, and Matthew M Williamson. 1998. Alternative essences of intelligence. AAAI/IAAI 1998(1998), 961–968.

Digital Library

[5]

David Byrne. 2021. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Quality & Quantity 1(2021), 1–22.

[6]

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. https://doi.org/10.1177/001316446002000104 arXiv:https://doi.org/10.1177/001316446002000104

[7]

David Conroy, Peta Wyeth, and Daniel Johnson. 2011. Modeling player-like behavior for game AI design. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology. ACM, New York, NY, 1–8.

Digital Library

[8]

William de Cothi, Nils Nyberg, Eva-Maria Griesbauer, Carole Ghanamé, Fiona Zisch, Julie Lefort, Lydia Fletcher, Charlotte Newton, Sophie Renaudineau, Daniel Bendor, Roddy Grieves, Éléonore Duvelle, Caswell Barry, and Hugo J. Spiers. 2020. Predictive Maps in Rats and Humans for Spatial Navigation. bioRxiv preprint: 2020.09.26.314815 1 (2020), 1–44.

[9]

Sam Devlin, Raluca Georgescu, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, and Katja Hofmann. 2021. Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation. In Proceedings of the 38th International Conference on Machine Learning. PMLR, Virtual, 1–10.

[10]

Hugh Durrant-Whyte and Tim Bailey. 2006. Simultaneous localization and mapping: part I. IEEE robotics & automation magazine 13, 2 (2006), 99–110.

[11]

Néstor García, Jan Rosell, and Raúl Suárez. 2017. Motion Planning by Demonstration With Human-Likeness Evaluation for Dual-Arm Robots. IEEE Transactions on Systems, Man, and Cybernetics: Systems 49, 11(2017), 2298–2307.

[12]

Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser. 2005. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 7052 (2005), 801–806.

[13]

Simon Hecker, Dengxin Dai, Alexander Liniger, Martin Hahner, and Luc Van Gool. 2020. Learning accurate and human-like driving using semantic maps and attention. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Piscataway, New Jersey, 2346–2353.

Digital Library

[14]

Philip Hingston. 2010. A new design for a turing test for bots. In Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games. IEEE, Piscataway, New Jersey, 345–350.

[15]

Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation. ACM, New York City, New York, 64–67.

Digital Library

[16]

Mikhail Jacob, Sam Devlin, and Katja Hofmann. 2020. “It’s Unwieldy and It Takes a Lot of Time”—Challenges and Opportunities for Creating Agents in Commercial Games. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 16. AAAI Press, Palo Alto, CA, 88–94.

[17]

Peter H. Kahn, Hiroshi Ishiguro, Batya Friedman, and Takayuki Kanda. 2006. What is a Human? - Toward Psychological Benchmarks in the Field of Human-Robot Interaction. In ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication, Vol. 1. IEEE, Piscatway, New Jersey, 364–371. https://doi.org/10.1109/ROMAN.2006.314461

[18]

Man-Je Kim, Kyung-Joong Kim, Seungjun Kim, and Anind K. Dey. 2018. Performance Evaluation Gaps in a Real-Time Strategy Game Between Human and Artificial Intelligence Players. IEEE Access 6(2018), 13575–13586. https://doi.org/10.1109/ACCESS.2018.2800016

[19]

John O’Keefe and Lynn Nadel. 1978. The hippocampus as a cognitive map. Oxford: Clarendon Press, New York City, New York.

[20]

Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5, 5 (2010), 411–419.

[21]

Ariel Rosenfeld, Moshe Cohen, Matthew E Taylor, and Sarit Kraus. 2018. Leveraging human knowledge in tabular reinforcement learning: A study of human subjects. The Knowledge Engineering Review 33 (2018), 1–26.

[22]

Matthias Scheutz, Paul Schermerhorn, James Kramer, and David Anderson. 2007. First steps toward natural human-like HRI. Autonomous Robots 22, 4 (2007), 411–423.

Digital Library

[23]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 1 (2017), 12 pages.

[24]

Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi. Advances in Neural Information Processing Systems 34 (2021), 28 pages.

[25]

Greg Snook. 2000. Simplified 3D movement and pathfinding using navigation meshes. Game programming gems 1, 1 (2000), 288–304.

[26]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press, 1 Broadway, Cambridge, MA.

Digital Library

[27]

Eric Wiewiora. 2010. Reward Shaping. Springer US, Boston, MA, 863–865. https://doi.org/10.1007/978-0-387-30164-8_731

Cited By

Pischedda DKaufmann VWudarczyk ORahman RHafner VKuhlen AHaynes J(2023)Human or AI? The brain knows it! A brain-based Turing Test to discriminate between human and artificial agents.2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309541(951-958)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309541
Momennejad I(2022)A rubric for human-like agents and NeuroAIPhilosophical Transactions of the Royal Society B: Biological Sciences10.1098/rstb.2021.0446378:1869Online publication date: 13-Dec-2022
https://doi.org/10.1098/rstb.2021.0446

Recommendations

Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

We aim to understand how people assess human likeness in navigation produced by people and artificially intelligent (AI) agents in a video game. To this end, we propose a novel AI agent with the goal of generating more human-like behavior. We collect ...
Modeling, Replicating, and Predicting Human Behavior: A Survey
Given the popular presupposition of human reasoning as the standard for learning and decision making, there have been significant efforts and a growing trend in research to replicate these innate human abilities in artificial systems. As such, topics ...
Synthesizing Explainable Behavior for Human-AI Collaboration
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems

As AI technologies enter our everyday lives at an ever increasing pace, there is a greater need for AI systems to work synergistically with humans. This requires AI systems to exhibit behavior that is explainable to humans. Synthesizing such behavior ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

3066 pages

ISBN:9781450391566

DOI:10.1145/3491101

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
1,491
Total Downloads

Downloads (Last 12 months)569
Downloads (Last 6 weeks)71

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pischedda DKaufmann VWudarczyk ORahman RHafner VKuhlen AHaynes J(2023)Human or AI? The brain knows it! A brain-based Turing Test to discriminate between human and artificial agents.2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309541(951-958)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309541
Momennejad I(2022)A rubric for human-like agents and NeuroAIPhilosophical Transactions of the Royal Society B: Biological Sciences10.1098/rstb.2021.0446378:1869Online publication date: 13-Dec-2022
https://doi.org/10.1098/rstb.2021.0446

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents