Work in Progress

Neural Language Models as What If? -Engines for HCI Research

Authors:

Perttu Hämäläinen,

Anton KunnariAuthors Info & Claims

IUI '22 Companion: Companion Proceedings of the 27th International Conference on Intelligent User Interfaces

Pages 77 - 80

https://doi.org/10.1145/3490100.3516458

Published: 22 March 2022 Publication History

Abstract

Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) and user experience (UX) research. In this poster paper, we explore and critically evaluate the potential of large-scale neural language models like GPT-3 in generating synthetic research data such as participant responses to interview questions. We observe that in the best case, GPT-3 can create plausible reflections of video game experiences and emotions, and adapt its responses to given demographic information. Compared to real participants, such synthetic data can be obtained faster and at a lower cost. On the other hand, the quality of generated data has high variance, and future work is needed to rigorously quantify the human-likeness, limitations, and biases of the models in the HCI domain.

References

[1]

Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old Wine in New Bottles or Novel Challenges: A Critical Analysis of Empirical Studies of User Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2689–2698. https://doi.org/10.1145/1978942.1979336

Digital Library

[2]

Julia Bopp, Jan Benjamin Vornhagen, Roosa Piitulainen, Barbara Keller, and Elisa D. Mekler. 2020. GamesAsArt. (July 2020). https://doi.org/10.17605/OSF.IO/RYVT6 Publisher: OSF.

[3]

Julia A. Bopp, Jan B. Vornhagen, and Elisa D. Mekler. 2021. ”My Soul Got a Little Bit Cleaner”: Art Experience in Videogames. Proc. ACM Hum.-Comput. Interact. 5, CHI PLAY, Article 237 (oct 2021), 19 pages. https://doi.org/10.1145/3474664

Digital Library

[4]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

[5]

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arxiv:2101.00027 [cs.CL]

[6]

David Ha and Jürgen Schmidhuber. 2018. World models. arXiv preprint arXiv:1803.10122(2018). http://arxiv.org/abs/1803.10122

[7]

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. 2020. Dream to Control: Learning Behaviors by Latent Imagination. arxiv:1912.01603 [cs.LG]

[8]

Jussi Jokinen, Aditya Acharya, Mohammad Uzair, Xinhui Jiang, and Antti Oulasvirta. 2021. Touchscreen Typing As Optimal Supervisory Control. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker (Eds.). ACM, 720:1–720:14. https://doi.org/10.1145/3411764.3445483

Digital Library

[9]

OpenAI. 2021. OpenAI GPT-3 playground. https://beta.openai.com/playground

[10]

Antti Oulasvirta. 2019. It’s time to rediscover HCI models. Interactions 26, 4 (2019), 52–56.

Digital Library

[11]

Ingrid Pettersson, Florian Lachner, Anna-Katharina Frison, Andreas Riener, and Andreas Butz. 2018. A Bermuda Triangle? A Review of Method Application and Triangulation in User Experience Evaluation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3173574.3174035

Digital Library

[12]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21(2020), 140:1–140:67. http://jmlr.org/papers/v21/20-074.html

[13]

Shaghayegh Roohi, Asko Relas, Jari Takatalo, Henri Heiskanen, and Perttu Hämäläinen. 2020. Predicting Game Difficulty and Churn Without Players. In CHI PLAY ’20: The Annual Symposium on Computer-Human Interaction in Play, Virtual Event, Canada, November 2-4, 2020, Pejman Mirza-Babaei, Victoria McArthur, Vero Vanden Abeele, and Max Birk (Eds.). ACM, 585–593. https://doi.org/10.1145/3410404.3414235

Digital Library

[14]

Jeff Sauro and James R Lewis. 2016. Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann.

[15]

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. 2020. Planning to Explore via Self-Supervised World Models. In Proceedings of the 37th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 8583–8592. https://proceedings.mlr.press/v119/sekar20a.html

Digital Library

[16]

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. arxiv:2102.09690 [cs.CL]

Cited By

Sweetser P(2024)Large Language Models and Video Games: A Preliminary Scoping ReviewProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665582(1-8)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665582
Zhou ZLi JZhang ZYu JDuh H(2024)Examining How the Large Language Models Impact the Conceptual Design with Human Designers: A Comparative Case StudyInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2370635(1-17)Online publication date: Jul-2024
https://doi.org/10.1080/10447318.2024.2370635
Lee WChun MJeong HJung H(2023)Toward Keyword Generation through Large Language ModelsCompanion Proceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581754.3584126(37-40)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581754.3584126
Show More Cited By

Index Terms

Neural Language Models as What If? -Engines for HCI Research

Index terms have been assigned to the content through auto-classification.

Recommendations

Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Collecting data is one of the bottlenecks of Human-Computer Interaction (HCI) research. Motivated by this, we explore the potential of large language models (LLMs) in generating synthetic user research data. We use OpenAI’s GPT-3 model to generate open-...
Towards Computational Support with Language Models for TTRPG Game Masters
FDG '23: Proceedings of the 18th International Conference on the Foundations of Digital Games

Tabletop role-playing games require game masters to mediate between the desires of a group of players and the affordances of a mechanically rich game world. In this paper, we explore the potential for large language models like GPT-3 to assist with the ...
A case study of intended versus actual experience of adaptivity in a tangible storytelling system
Abstract
This article presents a case study of an adaptive, tangible storytelling system called “The Reading Glove”. The research addresses a gap in the field of adaptivity for ubiquitous systems by taking a critical look at the notion of “adaptivity” and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

IUI '22 Companion: Companion Proceedings of the 27th International Conference on Intelligent User Interfaces

March 2022

142 pages

ISBN:9781450391450

DOI:10.1145/3490100

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 March 2022

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Conference

IUI '22

Sponsor:

IUI '22: 27th International Conference on Intelligent User Interfaces

March 22 - 25, 2022

Helsinki, Finland

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)6

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sweetser P(2024)Large Language Models and Video Games: A Preliminary Scoping ReviewProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665582(1-8)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665582
Zhou ZLi JZhang ZYu JDuh H(2024)Examining How the Large Language Models Impact the Conceptual Design with Human Designers: A Comparative Case StudyInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2370635(1-17)Online publication date: Jul-2024
https://doi.org/10.1080/10447318.2024.2370635
Lee WChun MJeong HJung H(2023)Toward Keyword Generation through Large Language ModelsCompanion Proceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581754.3584126(37-40)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581754.3584126
Hämäläinen PTavast MKunnari A(2023)Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case StudyProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580688(1-19)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580688

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents