research-article

Open access

Conversations with GUIs

Authors:

Daniel Buschek,

Antti OulasvirtaAuthors Info & Claims

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

Pages 1447 - 1457

https://doi.org/10.1145/3461778.3462124

Published: 28 June 2021 Publication History

All formats PDF

Abstract

Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N = 120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our findings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

Supplementary Material

VTT File (dis21-129-video_figure.vtt)

Download
1.16 KB

MP4 File (dis21-129-video_figure.mp4)

Supplemental video

Download
6.92 MB

References

[1]

Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2017. VQA: Visual Question Answering. Int. J. Comput. Vision (May 2017). https://doi.org/10.1007/s11263-016-0966-6

Digital Library

[2]

Tawfiq Ammari, Jofish Kaye, Janice Y. Tsai, and Frank Bentley. 2019. Music, Search, and IoT: How People (Really) Use Voice Assistants. ACM Trans. Comput.-Hum. Interact. 26, 3, Article 17 (April 2019). https://doi.org/10.1145/3311956

Digital Library

[3]

Richard A. Bolt. 1980. “Put-That-There”: Voice and Gesture at the Graphics Interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques (Seattle, USA) (SIGGRAPH ’80). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/800250.807503

Digital Library

[4]

Michael Braun, Anja Mainz, Ronee Chadowitz, Bastian Pfleging, and Florian Alt. 2019. At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland) (CHI ’19). Association for Computing Machinery, New York, USA. https://doi.org/10.1145/3290605.3300270

Digital Library

[5]

Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445762

Digital Library

[6]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3126594.3126651

Digital Library

[7]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2009.5206848

[8]

Umer Farooq, Jonathan Grudin, Ben Shneiderman, Pattie Maes, and Xiangshi Ren. 2017. Human Computer Integration versus Powerful Tools. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3027063.3051137

Digital Library

[9]

Ted Grover, Kael Rowan, Jina Suh, Daniel McDuff, and Mary Czerwinski. 2020. Design and Evaluation of Intelligent Agent Prototypes for Assistance with Focus and Productivity at Work. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3377325.3377507

Digital Library

[10]

Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, and Blaise Aguera y Arcas. 2020. ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces. In AAAI-21. https://arxiv.org/abs/2012.12350

[11]

Forrest Huang, Eldon Schoop, David Ha, and John Canny. 2020. Scones: Towards Conversational Authoring of Sketches. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3377325.3377485

Digital Library

[12]

Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland) (CHI ’19). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3290605.3300316

Digital Library

[13]

A. Baki Kocaballi, Juan C. Quiroz, Liliana Laranjo, Dana Rezazadegan, Rafal Kocielnik, Leigh Clark, Q. Vera Liao, Sun Young Park, Robert J. Moore, and Adam Miner. 2020. Conversational Agents for Health and Wellbeing. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3334480.3375154

Digital Library

[14]

Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2470654.2466420

Digital Library

[15]

Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2020. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, Article 9. https://doi.org/10.1145/3406324.3410710

Digital Library

[16]

Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445049

Digital Library

[17]

Yu-Tzu Lin and Morten Hertzum. 2020. How Do Designers Make User-Experience Design Decisions?. In Design, User Experience, and Usability. Interaction Design. Springer International Publishing, Cham.

[18]

Gesa Alena Linnemann and Regina Jucks. 2018. ‘Can I Trust the Spoken Dialogue System Because It Uses the Same Words as I Do?’—Influence of Lexically Aligned Spoken Dialogue Systems on Trustworthiness and User Satisfaction. Interacting with Computers 30, 3 (03 2018). https://doi.org/10.1093/iwc/iwy005 arXiv:https://academic.oup.com/iwc/article-pdf/30/3/173/24805335/iwy005.pdf

[19]

Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning Design Semantics for Mobile Apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3242587.3242650

Digital Library

[20]

Ying-Hsang Liu, Alexandre Arnold, Gérard Dupont, Catherine Kobus, and François Lancelot. 2020. Evaluation of conversational agents for aerospace domain. In Proc. CIRCLE.

[21]

Maria Muszynska, Denise Michels, and Emanuel von Zezschwitz. 2018. Not On My Phone: Exploring Users’ Conception of Related Permissions. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3170427.3188625

Digital Library

[22]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar. https://doi.org/10.3115/v1/D14-1162

[23]

Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, USA. https://doi.org/10.1145/3173574.3174214

Digital Library

[24]

Alireza Sahami Shirazi, Niels Henze, Albrecht Schmidt, Robin Goldberg, Benjamin Schmidt, and Hansjörg Schmauder. 2013. Insights into Layout Patterns of Mobile User Interfaces by an Automatic Analysis of Android Apps. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (London, United Kingdom) (EICS ’13). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2494603.2480308

Digital Library

[25]

Franco M. Segarra, Luis A. Leiva, and Roberto Paredes. 2011. A Relevant Image Search Engine with Late Fusion: Mixing the Roles of Textual and Visual Descriptors. In Proceedings of the 16th International Conference on Intelligent User Interfaces (Palo Alto, CA, USA) (IUI ’11). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1943403.1943496

Digital Library

[26]

Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents. Interactions 4, 6 (Nov. 1997). https://doi.org/10.1145/267505.267514

Digital Library

[27]

A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 12(2000). https://doi.org/10.1109/34.895972

Digital Library

[28]

Maartje ter Hoeve, Robert Sim, Elnaz Nouri, Adam Fourney, Maarten de Rijke, and Ryen W. White. 2020. Conversations with Documents: An Exploration of Document-Centered Assistance. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval(Vancouver BC, Canada) (CHIIR ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3343413.3377971

Digital Library

[29]

Sarah Theres Völkel, Renate Haeuslschmid, Anna Werner, Heinrich Hussmann, and Andreas Butz. 2020. How to Trick AI: Users’ Strategies for Protecting Themselves from Automatic Personality Assessment. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3313831.3376877

Digital Library

[30]

Sarah Theres Völkel, Penelope Kempf, and Heinrich Hussmann. 2020. Personalised Chats with Voice Assistants: The User Perspective. In Proceedings of the 2nd Conference on Conversational User Interfaces (Bilbao, Spain) (CUI ’20). Association for Computing Machinery, New York, NY, USA, Article 53. https://doi.org/10.1145/3405755.3406156

Digital Library

[31]

Sarah Theres Völkel, Christina Schneegass, Malin Eiband, and Daniel Buschek. 2020. What is ”Intelligent” in Intelligent User Interfaces? A Meta-Analysis of 25 Years of IUI. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3377325.3377500

Digital Library

[32]

Thiemo Wambsganss, Rainer Winkler, Matthias Söllner, and Jan Marco Leimeister. 2020. A Conversational Agent to Improve Response Quality in Course Evaluations. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3334480.3382805

Digital Library

[33]

Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445186

Digital Library

[34]

Michelle X. Zhou, Gloria Mark, Jingyi Li, and Huahai Yang. 2019. Trusting Virtual Agents: The Effect of Personality. ACM Trans. Interact. Intell. Syst. 9, 2–3, Article 10 (March 2019). https://doi.org/10.1145/3232077

Digital Library

Cited By

Abukadah HFereidouni MSiddique A(2024)Mapping Natural Language Intents to User Interfaces through Vision-Language Models2024 IEEE 18th International Conference on Semantic Computing (ICSC)10.1109/ICSC59802.2024.00045(237-244)Online publication date: 5-Feb-2024
https://doi.org/10.1109/ICSC59802.2024.00045
Ciarrone ALeiva LDubiel M(2024)Enricommender: Business Intelligence for User Interface DesignInteracting with Computers10.1093/iwc/iwae037Online publication date: 22-Aug-2024
https://doi.org/10.1093/iwc/iwae037
Lee SLaw MHoffman G(2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.235345141:2(1569-1584)Online publication date: 22-May-2024
https://doi.org/10.1080/10447318.2024.2353451
Show More Cited By

Recommendations

AI-Powered Health Chatbots: Toward a general architecture
Abstract
A Chatbot is a conversational agent that simulate human conversation, through text or voice messages. One of the first goals of a Chatbot is to interact with the user just like a human. When it comes to Health Chatbots, another main goal is to be ...
Investigating Paraphrasing-Based Data Augmentation for Task-Oriented Dialogue Systems
Text, Speech, and Dialogue
Abstract
With synthetic data generation, the required amount of human-generated training data can be reduced significantly. In this work, we explore the usage of automatic paraphrasing models such as GPT-2 and CVAE to augment template phrases for task-...
Loosely-coupled approach towards multi-modal browsing

Contemplating the concept of universal-access multi-modal browsing comes as one of the emerging "killer" technologies that promises broader and more flexible access to information, faster task completion, and advanced user experience. Inheriting the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

June 2021

2082 pages

ISBN:9781450384766

DOI:10.1145/3461778

Editors:
Wendy Ju
Jacobs Technion-Cornell Institute at Cornell Tech
,
Lora Oehlberg
University of Calgary
,
Sean Follmer
Stanford University
,
Sarah Fox
Carnegie Mellon University
,
Stacey Kuznetsov
Arizona State University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Academy of Finland

Conference

DIS '21

Sponsor:

SIGCHI

DIS '21: Designing Interactive Systems Conference 2021

June 28 - July 2, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Upcoming Conference

DIS '25

Sponsor:
sigchi

Designing Interactive Systems Conference

July 5 - 9, 2025

Funchal , Portugal

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
1,182
Total Downloads

Downloads (Last 12 months)248
Downloads (Last 6 weeks)41

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Abukadah HFereidouni MSiddique A(2024)Mapping Natural Language Intents to User Interfaces through Vision-Language Models2024 IEEE 18th International Conference on Semantic Computing (ICSC)10.1109/ICSC59802.2024.00045(237-244)Online publication date: 5-Feb-2024
https://doi.org/10.1109/ICSC59802.2024.00045
Ciarrone ALeiva LDubiel M(2024)Enricommender: Business Intelligence for User Interface DesignInteracting with Computers10.1093/iwc/iwae037Online publication date: 22-Aug-2024
https://doi.org/10.1093/iwc/iwae037
Lee SLaw MHoffman G(2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.235345141:2(1569-1584)Online publication date: 22-May-2024
https://doi.org/10.1080/10447318.2024.2353451
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Mottelson A(2023)Why do people take Screenshots on their Smartphones?Proceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596067(740-752)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596067
Lehmann F(2023)Mixed-Initiative Interaction with Computational Generative SystemsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3577061(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3577061
Jahanbakhsh FNouri ESim RWhite RFourney A(2022)Understanding Questions that Arise When Working with Business DocumentsProceedings of the ACM on Human-Computer Interaction10.1145/35557616:CSCW2(1-24)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555761
Feng SJiang MZhou TZhen YChen C(2022)Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI DevelopmentACM Transactions on Interactive Intelligent Systems10.1145/353106512:4(1-26)Online publication date: 4-Nov-2022
https://dl.acm.org/doi/10.1145/3531065

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten