research-article

Toward Value Scenario Generation Through Large Language Models

Authors:

Sungmin NaAuthors Info & Claims

CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing

Pages 212 - 220

https://doi.org/10.1145/3584931.3606960

Published: 14 October 2023 Publication History

Abstract

We propose a method of generating value scenarios for design research by leveraging ChatGPT, an AI-powered chatbot based on large language models. Identifying the needs of a vulnerable population, such as North Korean defectors, is challenging for researchers. To address this, we introduce ChatGPT-generated value scenarios, an extension of scenario-based design that supports critical, systemic, long-term thinking in current design practice, technology development, and deployment. Using our proposed method, we created a prompt to generate value scenarios on ChatGPT. Based on our analysis of the generated scenarios, we identified that ChatGPT could generate plausible information about Value Implications. However, it lacks details on Pervasiveness and Systemic Effects. After discussing the limitations and opportunities of ChatGPT in generating value scenarios, we conclude with suggestions for how ChatGPT might be better used to generate value scenarios.

References

[1]

Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate Muslims with violence. Nature Machine Intelligence 3, 6 (2021), 461–463. https://doi.org/10.1038/s42256-021-00359-2

[2]

Gerardo Adesso. 2022. GPT4: The ultimate brain. https://doi.org/10.22541/au.167052124.41804127/v2

[3]

Fatema AlDhaen. 2022. The Use of Artificial Intelligence in Higher Education – Systematic Review. Springer International Publishing, Cham, 269–285. https://doi.org/10.1007/978-3-031-13351-0_13

[4]

Abdulwahhab O. Alshammari and Hyunggu Jung. 2017. Designing community of practice systems: a value sensitive approach. In 2017 International Conference on Informatics, Health & Technology (ICIHT). IEEE, Riyadh, Saudi Arabia, 1–7. https://doi.org/10.1109/ICIHT.2017.7899005

[5]

Alissa Nicole Antle. 2006. Child-Personas: Fact or Fiction?. In Proceedings of the 6th Conference on Designing Interactive Systems (University Park, PA, USA) (DIS ’06). Association for Computing Machinery, New York, NY, USA, 22–30. https://doi.org/10.1145/1142405.1142411

Digital Library

[6]

Ömer Aydın and Enis Karaarslan. 2022. OpenAI ChatGPT generated literature review: Digital twin in healthcare. Emerging Computer Technologies 2 (2022), 22-41 pages. https://doi.org/10.2139/ssrn.4308687

[7]

Lanthao Benedikt, Chaitanya Joshi, Louisa Nolan, Ruben Henstra-Hill, Luke Shaw, and Sharon Hook. 2020. Human-in-the-Loop AI in Government: A Case Study. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 488–497. https://doi.org/10.1145/3377325.3377489

Digital Library

[8]

Margaret Cargill and Patrick O’Connor. 2021. Writing scientific research articles: Strategy and steps. John Wiley & Sons, Oxford, UK.

[9]

J.M. Carrol. 1999. Five reasons for scenario-based design. In Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers, Vol. Track3. IEEE, Maui, HI, USA, 123. https://doi.org/10.1109/HICSS.1999.772890

[10]

John M. Carroll. 2003. Scenario-Based Design. CRC Press, Florida, 45–70.

[11]

Yanran Chen and Steffen Eger. 2022. Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End. arxiv:2212.10522 [cs.CL]

[12]

Geoffrey M Currie. 2023. Academic integrity and artificial intelligence: is ChatGPT hype, hero or heresy?https://doi.org/10.1053/j.semnuclmed.2023.04.008

[13]

Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 2020. Human-in-the-loop Artificial Intelligence for Fighting Online Misinformation: Challenges and Opportunities.IEEE Data Eng. Bull. 43, 3 (2020), 65–74.

[14]

Jianyang Deng and Yijia Lin. 2023. The Benefits and Challenges of ChatGPT: An Overview. Frontiers in Computing and Intelligent Systems 2, 2 (Jan. 2023), 81–83. https://doi.org/10.54097/fcis.v2i2.4465

[15]

Michael Dowling and Brian Lucey. 2023. ChatGPT for (finance) research: The Bananarama conjecture. Finance Research Letters 53 (2023), 103662. https://doi.org/10.1016/j.frl.2023.103662

[16]

Batya Friedman, Peter H Kahn, Alan Borning, and Alina Huldtgren. 2013. Value sensitive design and information systems. Early engagement and new technologies: Opening up the laboratory 5 (2013), 55–95. https://doi.org/10.1007/978-94-007-7844-3_4

[17]

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. arxiv:2209.07858 [cs.CL]

[18]

Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. 2023. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digital Medicine 6, 1 (2023), 75. https://doi.org/10.1038/s41746-023-00819-6

[19]

Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social Biases in NLP Models as Barriers for Persons with Disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5491–5501. https://doi.org/10.18653/v1/2020.acl-main.487

[20]

Andrew Ng Isa Fulford. 2023. ChatGPT Prompt Engineering for Developers. DeepLearning.AI. https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/Accessed: May 3, 2023.

[21]

Hyunggu Jung, Woosuk Seo, and Michelle Cha. 2017. Personas and Scenarios to Design Technologies for North Korean Defectors with Depression. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17 Companion). Association for Computing Machinery, New York, NY, USA, 215–218. https://doi.org/10.1145/3022198.3026308

Digital Library

[22]

David Jungwirth and Daniela Haluza. 2023. Artificial Intelligence and Public Health: An Exploratory Study. International Journal of Environmental Research and Public Health 20, 5 (Mar 2023), 4541. https://doi.org/10.3390/ijerph20054541

[23]

HA Kolnick, Jennifer E Miller, Olivia Dupree, and Lisa Gualtieri. 2021. Design Thinking to Create a Remote Patient Monitoring Platform for Older Adults’ Homes. https://doi.org/10.5210/ojphi.v13i1.11582

[24]

Wanhae Lee, Minki Chun, Hyeonhak Jeong, and Hyunggu Jung. 2023. Toward Keyword Generation through Large Language Models. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 37–40. https://doi.org/10.1145/3581754.3584126

Digital Library

[25]

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 563–569. https://doi.org/10.1145/3545945.3569770

Digital Library

[26]

Ana Isabel Martins, Alexandra Queirós, Nelson Pacheco Rocha, Telmo Neves, António Damasceno, and Luisa Arieira. 2018. Personas and Scenarios to Improve the Development of an Electronic Social Record Platform. In Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion (Thessaloniki, Greece) (DSAI ’18). Association for Computing Machinery, New York, NY, USA, 344–351. https://doi.org/10.1145/3218585.3218681

Digital Library

[27]

Lisa P. Nathan, Predrag V. Klasnja, and Batya Friedman. 2007. Value Scenarios: A Technique for Envisioning Systemic Effects of New Technologies. In CHI ’07 Extended Abstracts on Human Factors in Computing Systems (San Jose, CA, USA) (CHI EA ’07). Association for Computing Machinery, New York, NY, USA, 2585–2590. https://doi.org/10.1145/1240866.1241046

Digital Library

[28]

Jin-Won Noh and So Hee Lee. 2020. Trauma history and mental health of North Korean defectors. Current Behavioral Neuroscience Reports 7 (2020), 250–257. https://doi.org/10.1007/s40473-020-00219-0

[29]

Ministry of Unification Settlement support for North Korean defectors. 2022. 1. Number of North Korean defectors entering South Korea. https://www.unikorea.go.kr/eng_unikorea/whatwedo/support/. Accessed: July 3, 2023.

[30]

OpenAI. 2022. ChatGPT Prompt Design. OpenAI. https://platform.openai.com/docs/guides/completion/prompt-designAccessed: May 3, 2023.

[31]

OpenAI. 2023. ChatGPT: Optimizing Language Models for Dialogue. OpenAI. https://chat.openai.com/Accessed: May 3, 2023.

[32]

Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare 11, 6 (2023), 887 pages. https://doi.org/10.3390/healthcare11060887

[33]

Anna Stone. 2022. Student perceptions of academic integrity: a qualitative study of understanding, consequences, and impact. Journal of Academic Ethics (2022), 19 pages. https://doi.org/10.1007/s10805-022-09461-5

[34]

Raju Vaishya, Anoop Misra, and Abhishek Vaish. 2023. ChatGPT: Is this version good for healthcare and research?Diabetes & Metabolic Syndrome: Clinical Research & Reviews 17, 4 (2023), 102744. https://doi.org/10.1016/j.dsx.2023.102744

[35]

Liansheng Wang, Lianyu Zhou, Wenxian Yang, and Rongshan Yu. 2022. Deepfakes: a new threat to image fabrication in scientific publications?Patterns 3, 5 (2022), 4 pages. https://doi.org/10.1016/j.patter.2022.100509

[36]

Pontus Wärnestål, Petra Svedberg, and Jens Nygren. 2014. Co-Constructing Child Personas for Health-Promoting Services with Vulnerable Children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3767–3776. https://doi.org/10.1145/2556288.2557115

Digital Library

[37]

Karsten Wenzlaff and Sebastian Spaeth. 2022. Smarter than Humans? Validating how OpenAI’s ChatGPT model explains Crowdfunding, Alternative Finance and Community Finance.WiSo-HH Working Paper Series 75. University of Hamburg, Faculty of Business, Economics and Social Sciences, WISO Research Laboratory. https://doi.org/10.2139/ssrn.4302443

[38]

Michele A. Williams, Amy Hurst, and Shaun K. Kane. 2013. "Pray before You Step out": Describing Personal and Situational Blind Navigation Behaviors. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (Bellevue, Washington) (ASSETS ’13). Association for Computing Machinery, New York, NY, USA, Article 28, 8 pages. https://doi.org/10.1145/2513383.2513449

Digital Library

[39]

Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23 Companion). Association for Computing Machinery, New York, NY, USA, 75–78. https://doi.org/10.1145/3581754.3584136

Digital Library

[40]

J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/3544548.3581388

Digital Library

[41]

Xiaoming Zhai. 2022. ChatGPT user experience: Implications for education. https://doi.org/10.2139/ssrn.4312418

[42]

Jianlong Zhou, Heimo Müller, Andreas Holzinger, and Fang Chen. 2023. Ethical ChatGPT: Concerns, Challenges, and Commandments. arxiv:2305.10646 [cs.AI]

Cited By

Gao JGebreegziabher SChoo KLi TPerrault SMalone T(2024)A Taxonomy for Human-LLM Interaction Modes: An Initial ExplorationExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650786(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650786

Index Terms

Toward Value Scenario Generation Through Large Language Models
1. Human-centered computing
  1. Interaction design
    1. Systems and tools for interaction design

Recommendations

Value scenarios: a technique for envisioning systemic effects of new technologies
CHI EA '07: CHI '07 Extended Abstracts on Human Factors in Computing Systems

In this paper we argue that there is a scarcity of methods which support critical, systemic, long-term thinking in current design practice, technology development and deployment. To address this need we introduce value scenarios, an extension of ...
Science fiction as a value scenario for historical technology
Abstract
The value scenario is a useful tool in the sheaf of methods within value sensitive design. When envisioning new technology, this tool supports the designer in speculatively considering relevant stakeholders, values expressed or rebuffed by an ...
Alternative/Exceptional Scenario Generation with Differential Scenario
ICSEA '08: Proceedings of the 2008 The Third International Conference on Software Engineering Advances

A method of generating alternative/exceptional scenarios using differential information is presented. Behaviors of normal scenarios of similar purpose are quite similar, while actors and data in scenarios are different among these scenarios. We derive ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing

October 2023

596 pages

ISBN:9798400701290

DOI:10.1145/3584931

Editors:
Casey Fiesler
University of Colorado Boulder, USA
,
Loren Terveen
University of Minnesota, USA
,
Morgan Ames
University of California, Berkeley, USA
,
Susan Fussell
Cornell University, USA
,
Eric Gilbert
University of Michigan, USA
,
Vera Liao
Microsoft Research Montreal, Canada
,
Xiaojuan Ma
Hong Kong University of Science and Technology, Hong Kong
,
Xinru Page
Brigham Young University, USA
,
Mark Rouncefield
Lancaster University, UK
,
Vivek Singh
Rutgers University, USA
,
Pamela Wisniewski
Vanderbilt University, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CSCW '23

Sponsor:

SIGCHI

CSCW '23: Computer Supported Cooperative Work and Social Computing

October 14 - 18, 2023

MN, Minneapolis, USA

Acceptance Rates

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
438
Total Downloads

Downloads (Last 12 months)438
Downloads (Last 6 weeks)25

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao JGebreegziabher SChoo KLi TPerrault SMalone T(2024)A Taxonomy for Human-LLM Interaction Modes: An Initial ExplorationExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650786(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650786

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents