research-article

AI Knowledge: Improving AI Delegation through Human Enablement

Authors:

Alexander BenlianAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 25, Pages 1 - 17

https://doi.org/10.1145/3544548.3580794

Published: 19 April 2023 Publication History

Abstract

When collaborating with artificial intelligence (AI), humans can often delegate tasks to leverage complementary AI competencies. However, humans often delegate inefficiently. Enabling humans with knowledge about AI can potentially improve inefficient AI delegation. We conducted a between-subjects experiment (two groups, n = 111) to examine how enabling humans with AI knowledge can improve AI delegation in human-AI collaboration. We find that AI knowledge-enabled humans align their delegation decisions more closely with their assessment of how suitable a task is for humans or AI (i.e., task appraisal). We show that delegation decisions closely aligned with task appraisal increase task performance. However, we also find that AI knowledge lowers future intentions to use AI, suggesting that AI knowledge is not strictly positive for human-AI collaboration. Our study contributes to HCI design guidelines with a new perspective on AI features, educating humans regarding general AI functioning and their own (human) performance and biases.

Supplementary Material

MP4 File (3544548.3580794-talk-video.mp4)

Pre-recorded Video Presentation

Download
235.62 MB

References

[1]

Martin Adam, Konstantin Roethke and Alexander Benlian. 2022. Human Versus Automated Sales Agents: How and Why Customer Responses Shift Across Sales Stages. Information Systems Research, Forthcoming (2022). https://doi.org/10.1287/isre.2022.1171

Digital Library

[2]

Pär J. Ågerfalk. 2020. Artificial intelligence as digital agency. European Journal of Information Systems, 29, 1 (2020), 1-8. https://doi.org/10.1080/0960085x.2020.1721947

[3]

Amazon. 2017. Tutorial: How to label thousands of images using the crowd. Retrieved August 10, 2022 from https://blog.mturk.com/tutorial-how-to-label-thousands-of-images-using-the-crowd-bea164ccbefc

[4]

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300233

Digital Library

[5]

Eduard Anton, Alina Behne and Frank Teuteberg. 2020. The humans behind artificial intelligence - An operationalisation of AI competencies. In Proceedings of the European Conference on Information Systems. Virtual. https://aisel.aisnet.org/ecis2020_rp/141

[6]

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila and Francisco Herrera. 2020. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58 (2020), 82-115. https://doi.org/10.1016/j.inffus.2019.12.012

Digital Library

[7]

Sangeeta Bharadwaj Badal and Bryant Ott. 2015. Delegating: A Huge Management Challenge for Entrepreneurs. Retrieved August 10, 2022 from https://news.gallup.com/businessjournal/182414/delegating-huge-management-challenge-entrepreneurs.aspx

[8]

Aaron Baird and Likoebe M. Maruping. 2021. The Next Generation of Research on IS Use: A Theoretical Framework of Delegation to and from Agentic IS Artifacts. MIS Quarterly, 45, 1 (2021), 315-341. https://doi.org/10.25300/misq/2021/15882

[9]

Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S. Weld, Walter S. Lasecki and Eric Horvitz. 2019. Update in Human-AI Team. Understanding and Addressing the performance compatibility tradeoff. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19). Honolulu, USA.

[10]

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445717

Digital Library

[11]

Reuben M. Baron and David A. Kenny. 1986. The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol, 51, 6 (1986), 1173-1182. https://doi.org/10.1037//0022-3514.51.6.1173

[12]

Geneviève Bassellier, Izak Benbasat and Blaize Horner Reich. 2003. The Influence of Business Managers' IT Competence on Championing IT. Information Systems Research, 14, 4 (2003), 317-336. https://doi.org/10.1287/isre.14.4.317.24899

Digital Library

[13]

Geneviève Bassellier, Blaize Horner Reich and Izak Benbasat. 2015. Information Technology Competence of Business Managers: A Definition and Research Model. Journal of Management Information Systems, 17, 4 (2015), 159-182. https://doi.org/10.1080/07421222.2001.11045660

Digital Library

[14]

Kevin Bauer, Moritz von Zahn and Oliver Hinz. Forthcoming. Expl(AI)ned: The Impact of Explainable Articial Intelligence on Users' Information Processing. Information Systems Research (Forthcoming). https://doi.org/10.1287/xxxx.0000.0000

[15]

Anne Beaudry and Alain Pinsonneault. 2005. Understanding User Responses to Information Technology: A Coping Model of User Adaptation. MIS Quarterly, 29, 3 (2005). https://doi.org/10.2307/25148693

[16]

Hind Benbya, Stella Pachidi and Sirkka L. Jarvenpaa. 2021. Special Issue Editorial: Artificial Intelligence in Organizations: Implications for Information Systems Research. Journal of the Association for Information Systems, 22, 2 (2021), 281-303. https://doi.org/10.17705/1jais.00662

[17]

Nicholas Berente, Bin Gu, Jan Recker and Radhika Santhanam. 2021. Managing Artificial Intelligence. MIS Quarterly, 45, 3 (2021), 1433-1450. https://doi.org/10.25300/MISQ/2021/16274

[18]

Anol Bhattacherjee. 2001. Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly, 25, 3 (2001), 351-370. https://doi.org/10.2307/3250921

Digital Library

[19]

Anol Bhattacherjee, Christopher J. Davis, Amy J. Connolly, Neset Hikmet, Frantz Rowe and Régis Meissonier. 2017. User response to mandatory IT use: a coping theory perspective. European Journal of Information Systems, 27, 4 (2017), 395-414. https://doi.org/10.1057/s41303-017-0047-0

[20]

Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe and Michael Terry. 2019. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300234

Digital Library

[21]

Wanling Cai, Yucheng Jin and Li Chen. 2022. Impacts of Personal Characteristics on User Trust in Conversational Recommender Systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3517471

Digital Library

[22]

Longbing Cao. 2020. AI in Finance: A Review. SSRN Electronic Journal (2020). https://doi.org/10.2139/ssrn.3647625

[23]

Noel Carroll. 2021. Augmented Intelligence: An Actor-Network Theory Perspective. In Proceedings of the European Conference on Information Systems. Marrakesh, Morocco. https://aisel.aisnet.org/ecis2021_rp/37

[24]

Dilek Cetindamar, Kirsty Kitto, Mengjia Wu, Yi Zhang, Babak Abedin and Simon Knight. 2022. Explicating AI Literacy of Employees at Digital Workplaces. IEEE Transactions on Engineering Management (2022), 1-14. https://doi.org/10.1109/tem.2021.3138503

[25]

Hyesun Choung, Prabu David and Arun Ross. 2022. Trust in AI and Its Role in the Acceptance of AI Technologies. International Journal of Human–Computer Interaction (2022), 1-13. https://doi.org/10.1080/10447318.2022.2050543

[26]

Clifford C. Clogg, Eva Petkova and Adamantios Haritou. 1995. Statistical Methods for Comparing Regression Coefficients Between Models. American Journal of Sociology, 100, 5 (1995), 1261-1293. https://doi.org/10.1086/230638

[27]

Jacob Cohen, Patricia Cohen, Stephan G. West and Leona S. Aiken. 2003. Applied multiple regression/correlation analysis for the behavioral sciences. Erlbaum. Hillsdale, NJ

[28]

W. Alec Cram, Jeffrey G. Proudfoot and John D'Arcy. 2020. When enough is enough: Investigating the antecedents and consequences of information security fatigue. Information Systems Journal, 31, 4 (2020), 521-549. https://doi.org/10.1111/isj.12319

[29]

Dominik Dellermann, Philipp Ebel, Matthias Söllner and Jan Marco Leimeister. 2019. Hybrid Intelligence. Business & Information Systems Engineering, 61, 5 (2019), 637-643. https://doi.org/10.1007/s12599-019-00595-2

[30]

Xianghua Ding, Yanqi Jiang, Xiankang Qin, Yunan Chen, Wenqiang Zhang and Lizhe Qi. 2019. Reading Face, Reading Health. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300435

Digital Library

[31]

Mateusz Dolata, Stefan Feuerriegel and Gerhard Schwabe. 2021. A sociotechnical view of algorithmic fairness. Information Systems Journal, 32, 4 (2021), 754-818. https://doi.org/10.1111/isj.12370

[32]

Philipp Ebel, Matthias Söllner, Jan Marco Leimeister, Kevin Crowston and Gert-Jan de Vreede. 2021. Hybrid intelligence in business networks. Electronic Markets, 31, 2 (2021), 313-318. https://doi.org/10.1007/s12525-021-00481-4

[33]

Christian Engel, Philipp Ebel and Jan Marco Leimeister. 2022. Cognitive automation. Electronic Markets, 32, 1 (2022), 339-350. https://doi.org/10.1007/s12525-021-00519-7

[34]

Yoram Eshet-Alkalai. 2004. Digital literacy. Journal of Educational Multimedia and Hypermedia, 13, 1 (2004), 93-106.

[35]

Susan Folkman. 2013. Stress: Appraisal and Coping. In Encyclopedia of Behavioral Medicine. M. D. Gellman and J. R. Turner (Eds.). Springer. New York.

[36]

Claes Fornell and David F. Larcker. 1981. Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research, 18, 1 (1981). https://doi.org/10.2307/3151312

[37]

Andreas Fügener, Jörn Grahl, Alok Gupta and Wolfgang Ketter. 2021. Cognitive Challenges in Human–Artificial Intelligence Collaboration: Investigating the Path Toward Productive Delegation. Information Systems Research (2021). https://doi.org/10.1287/isre.2021.1079

Digital Library

[38]

Andreas Fügener, Jörn Grahl, Alok Gupta and Wolfgang Ketter. 2021. Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI. MIS Quarterly, 45, 3 (2021), 1527-1556. https://doi.org/10.25300/misq/2021/16553

[39]

Google. 2022. Advanced Guide to Inception v3. Retrieved July 12, 2022 from https://cloud.google.com/tpu/docs/inception-v3-advanced

[40]

Sophia Hadash, Martijn C. Willemsen, Chris Snijders and Wijnand A. Ijsselsteijn. 2022. Improving understandability of feature contributions in model-agnostic explainable AI tools. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3517650

Digital Library

[41]

Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch and Jeffrey P. Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174023

Digital Library

[42]

Teresa Heyder and Oliver Posegga. 2021. Extending the foundations of AI literacy. In Proceedings of the International Conference on Information Systems. Austin, USA. https://aisel.aisnet.org/icis2021/is_future_work/is_future_work/9

[43]

Bengt Holmström. 1979. Moral Hazard and Observability. The Bell Journal of Economics, 10, 1 (1979). https://doi.org/10.2307/3003320

[44]

Kristina Höök. 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers, 12, 4 (2000), 409-426. https://doi.org/10.1016/s0953-5438(99)00006-5

[45]

Qian Hu, Yaobin Lu, Zhao Pan, Yeming Gong and Zhilin Yang. 2021. Can AI artifacts influence human cognition? The effects of artificial autonomy in intelligent personal assistants. International Journal of Information Management, 56 (2021). https://doi.org/10.1016/j.ijinfomgt.2020.102250

Digital Library

[46]

Jacob Jacoby. 1984. Perspectives on Information Overload. Journal of Consumer Research, 10, 4 (1984). https://doi.org/10.1086/208981

[47]

Hemant Jain, Balaji Padmanabhan, Paul A. Pavlou and T. S. Raghu. 2021. Editorial for the Special Section on Humans, Algorithms, and Augmented Intelligence: The Future of Work, Organizations, and Society. Information Systems Research, 32, 3 (2021), 675-687. https://doi.org/10.1287/isre.2021.1046

Digital Library

[48]

Ekaterina Jussupow, Kai Spohrer, Armin Heinzl and Joshua Gawlitza. 2021. Augmenting Medical Diagnosis Decisions? An Investigation into Physicians’ Decision-Making Process with Artificial Intelligence. Information Systems Research, 32, 3 (2021), 713-735. https://doi.org/10.1287/isre.2020.0980

Digital Library

[49]

Daniel Kahneman and Amos Tversky. 1984. Choices, values, and frames. American Psychologist, 39, 4 (1984), 341-350. https://doi.org/10.1037/0003-066x.39.4.341

[50]

Georgi Dimov Kerpedzhiev, Ulrich Matthias König, Maximilian Röglinger and Michael Rosemann. 2020. An Exploration into Future Business Process Management Capabilities in View of Digitalization. Business & Information Systems Engineering, 63, 2 (2020), 83-96. https://doi.org/10.1007/s12599-020-00637-0

[51]

Alison King. 1992. Comparison of Self-Questioning, Summarizing, and Notetaking-Review as Strategies for Learning From Lectures. American Educational Research Journal, 29, 2 (1992), 303-323. https://doi.org/10.3102/00028312029002303

[52]

Rafal Kocielnik, Saleema Amershi and Paul N. Bennett. 2019. Will You Accept an Imperfect AI? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300641

Digital Library

[53]

Vivian Lai, Samuel Carton, Rajat Bhatnagar, Q. Vera Liao, Yunfeng Zhang and Chenhao Tan. 2022. Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3501999

Digital Library

[54]

Richard S. Lazarus and Susan Folkman. 1984. Stress, appraisal and coping. Springer. New York

[55]

Carrie R. Leana. 1986. Predictors and Consequences of Delegation. Academy of Management Journal, 29, 4 (1986), 754-774. https://doi.org/10.2307/255943

[56]

Carrie R. Leana. 1987. Power relinquishment versus power sharing: Theoretical clarification and empirical comparison of delegation and participation. Journal of Applied Psychology, 72, 2 (1987), 228-233. https://doi.org/10.1037/0021-9010.72.2.228

[57]

Minsun Lee and Jee‐Sun Park. 2022. Do parasocial relationships and the quality of communication with AI shopping chatbots determine middle‐aged women consumers' continuance usage intentions? Journal of Consumer Behaviour, 21, 4 (2022), 842-854. https://doi.org/10.1002/cb.2043

[58]

Claire Liang, Julia Proft, Erik Andersen and Ross A. Knepper. 2019. Implicit Communication of Actionable Information in Human-AI teams. In Proceedings of the Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300325

Digital Library

[59]

Q. Vera Liao, Daniel Gruen and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376590

Digital Library

[60]

Duri Long and Brian Magerko. 2020. What is AI Literacy? Competencies and Design Considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu, USA.

Digital Library

[61]

Brian Lubars and Chenhao Tan. 2019. Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability. In Proceedings of the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada.

[62]

Scott B. MacKenzie, Philip M. Podsakoff and Nathan P. Podsakoff. 2011. Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques. MIS Quarterly, 35, 2 (2011). https://doi.org/10.2307/23044045

[63]

Naresh K. Malhotra. 1982. Information Load and Consumer Decision Making. Journal of Consumer Research, 8, 4 (1982). https://doi.org/10.1086/208882

[64]

Connor J. McCabe, Dale S. Kim and Kevin M. King. 2018. Improving Present Practices in the Visual Display of Interactions. Adv Methods Pract Psychol Sci, 1, 2 (2018), 147-165. https://doi.org/10.1177/2515245917746792

[65]

Nathan J. McNeese, Beau G. Schelble, Lorenzo Barberis Canonico and Mustafa Demir. 2021. Who/What Is My Teammate? Team Composition Considerations in Human–AI Teaming. IEEE Transactions on Human-Machine Systems, 51, 4 (2021), 288-299. https://doi.org/10.1109/thms.2021.3086018

[66]

Patrick Mikalef, Kieran Conboy, Jenny Eriksson Lundström and Aleš Popovič. 2022. Thinking responsibly about responsible AI and ‘the dark side’ of AI. European Journal of Information Systems, 31, 3 (2022), 257-268. https://doi.org/10.1080/0960085x.2022.2026621

[67]

Allen E. Milewski and Steven H. Lewis. 1997. Delegating to software agents. International Journal of Human-Computer Studies, 46, 4 (1997), 485-500. https://doi.org/10.1006/ijhc.1996.0100

Digital Library

[68]

Franklin G. Moore. 1982. The Management of Organizations. Wiley. New York

[69]

Alex Murray, Jen Rhymer and David G. Sirmon. 2021. Humans and Technology: Forms of Conjoined Agency in Organizations. Academy of Management Review, 46, 3 (2021), 552-571. https://doi.org/10.5465/amr.2019.0186

[70]

Jordan Navarro, Sarah Allali, Nicolas Cabrignac and Julien Cegarra. 2021. Impact of Pilot's Expertise on Selection, Use, Trust, and Acceptance of Automation. IEEE Transactions on Human-Machine Systems, 51, 5 (2021), 432-441. https://doi.org/10.1109/thms.2021.3090199

[71]

Franz Neyer, Juliane Felber and Claudia Gebhardt. 2016. Kurzskala zur Erfassung von Technikbereitschaft (technology commitment). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS) (2016). https://doi.org/10.6102/zis244

[72]

Davy Tsz Kit Ng, Jac Ka Lok Leung, Samuel Kai Wah Chu and Maggie Shen Qiao. 2021. Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2 (2021). https://doi.org/10.1016/j.caeai.2021.100041

[73]

Andy Nguyen, Yvonne Hong and Lesley A. Gardner. 2020. A Taxonomy of Digital Learning Activities for Digital Inclusion. In Proceedings of the European Conference on Information Systems. Virtual. https://aisel.aisnet.org/ecis2020_rp/135

[74]

Tatsuya Nomura, Tomohiro Suzuki, Takayuki Kanda and Kensuke Kato. 2006. Measurement of negative attitudes toward robots. Interaction Studies, 7, 3 (2006), 437-454. https://doi.org/10.1075/is.7.3.14nom

[75]

Ikujiro Nonaka. 1994. A dynamic theory of organizational knowledge creation. Organization Science, 5, 1 (1994), 14-37. http://www.jstor.com/stable/2635068

Digital Library

[76]

Milda Norkute, Nadja Herger, Leszek Michalak, Andrew Mulder and Sally Gao. 2021. Towards Explainable AI: Assessing the Usefulness and Impact of Added Explainability Features in Legal Document Summarization. In Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411763.3443441

Digital Library

[77]

Philip M. Podsakoff, Scott B. MacKenzie and Nathan P. Podsakoff. 2016. Recommendations for Creating Better Concept Definitions in the Organizational, Behavioral, and Social Sciences. Organizational Research Methods, 19, 2 (2016), 159-203. https://doi.org/10.1177/1094428115624965

[78]

Qing Rao and Jelena Frtunikj. 2018. Deep learning for self-driving cars. In Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems. Gothenburg, Sweden.

Digital Library

[79]

Sebastian Schuetz and Viswanath Venkatesh. 2020. Research Perspectives: The Rise of Human Machines: How Cognitive Computing Systems Challenge Assumptions of User-System Interaction. Journal of the Association for Information Systems (2020), 460-482. https://doi.org/10.17705/1jais.00608

[80]

Axel Schulte, Diana Donath and Douglas S. Lange. 2016. Design Patterns for Human-Cognitive Agent Teaming. In Engineering Psychology and Cognitive Ergonomics. D. Harris (Ed.). Springer. Cham.

[81]

Haifeng Shen, Kewen Liao, Zhibin Liao, Job Doornberg, Maoying Qiao, Anton van den Hengel and Johan W. Verjans. 2021. Human-AI Interactive and Continuous Sensemaking: A Case Study of Image Classification using Scribble Attention Maps. In Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411763.3451798

Digital Library

[82]

Herbert A. Simon. 1990. Bounded rationality. In Utility and Probability. J. Eatwell, M. Milgate and P. Newman (Eds.). Palgrave Macmillan. London.

[83]

Ida Someh, Michael Davern, Christoph F. Breidbach and Graeme Shanks. 2019. Ethical Issues in Big Data Analytics: A Stakeholder Perspective. Communications of the Association for Information Systems (2019), 718-747. https://doi.org/10.17705/1cais.04434

[84]

Stanford Vision Labs. 2022. ImageNet. Retrieved July 12, 2022 from https://www.image-net.org/index.php

[85]

Frederik Stjernfelt and Anne Mette Lauritzen. 2020. Facebook's Handbook of Content Removal. In Your Post Has Been Removed. Springer. Cham.

[86]

Yi Sun, Shihui Li and Lingling Yu. 2021. The dark sides of AI personal assistant: effects of service failure on user continuance intention. Electronic Markets, 32, 1 (2021), 17-39. https://doi.org/10.1007/s12525-021-00483-2

[87]

Thomas Süße, Maria Kobert and Caroline Kries. 2021. Antecedents of Constructive Human-AI Collaboration: An Exploration of Human Actors’ Key Competencies. In Smart and Sustainable Collaborative Networks 4.0. p. 113-124.

[88]

Monideepa Tarafdar, Xinru Page and Marco Marabelli. 2022. Algorithms as co‐workers: Human algorithm role interactions in algorithmic work. Information Systems Journal (2022). https://doi.org/10.1111/isj.12389

[89]

Mike Teodorescu, Lily Morse, Yazeed Awwad and Gerald Kane. 2021. Failures of Fairness in Automation Require a Deeper Understanding of Human-ML Augmentation. MIS Quarterly, 45, 3 (2021), 1483-1500. https://doi.org/10.25300/misq/2021/16535

[90]

Scott Thiebes, Sebastian Lins and Ali Sunyaev. 2020. Trustworthy artificial intelligence. Electronic Markets, 31, 2 (2020), 447-464. https://doi.org/10.1007/s12525-020-00441-4

[91]

James Y. L. Thong, Se-Joon Hong and Kar Yan Tam. 2006. The effects of post-adoption beliefs on the expectation-confirmation model for information technology continuance. International Journal of Human-Computer Studies, 64, 9 (2006), 799-810. https://doi.org/10.1016/j.ijhcs.2006.05.001

Digital Library

[92]

Güliz Tokadlı, Michael C. Dorneich and Michael Matessa. 2021. Evaluation of Playbook Delegation Approach in Human-Autonomy Teaming for Single Pilot Operations. International Journal of Human–Computer Interaction, 37, 7 (2021), 703-716. https://doi.org/10.1080/10447318.2021.1890485

[93]

Eric J. Topol. 2019. High-performance medicine: the convergence of human and artificial intelligence. Nat Med, 25, 1 (Jan 2019), 44-56. https://doi.org/10.1038/s41591-018-0300-7

[94]

P. Tschandl, C. Rinner, Z. Apalla, G. Argenziano, N. Codella, A. Halpern, M. Janda, A. Lallas, C. Longo, J. Malvehy, J. Paoli, S. Puig, C. Rosendahl, H. P. Soyer, I. Zalaudek and H. Kittler. 2020. Human-computer collaboration for skin cancer recognition. Nat Med, 26, 8 (2020), 1229-1234. https://doi.org/10.1038/s41591-020-0942-0

[95]

Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science, 185, 4157 (1974), 1124–1131. http://www.jstor.org/stable/1738360

[96]

Benjamin van Giffen, Dennis Herhausen and Tobias Fahse. 2022. Overcoming the pitfalls and perils of algorithms: A classification of machine learning biases and mitigation methods. Journal of Business Research, 144 (2022), 93-106. https://doi.org/10.1016/j.jbusres.2022.01.076

[97]

Jhim Kiel M. Verame, Enrico Costanza, Joel Fischer, Andy Crabtree, Sarvapali D. Ramchurn, Tom Rodden and Nicholas R. Jennings. 2018. Learning from the Veg Box. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174021

Digital Library

[98]

Michael Vössing, Niklas Kühl, Matteo Lind and Gerhard Satzger. 2022. Designing Transparency for Effective Human-AI Collaboration. Information Systems Frontiers, 24, 3 (2022), 877-895. https://doi.org/10.1007/s10796-022-10284-3

Digital Library

[99]

Bingcheng Wang, Pei-Luen Patrick Rau and Tianyi Yuan. 2022. Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology (2022), 1-14. https://doi.org/10.1080/0144929x.2022.2072768

[100]

Danding Wang, Qian Yang, Ashraf Abdul and Brian Y. Lim. 2019. Designing Theory-Driven User-Centric Explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300831

Digital Library

[101]

Leslie Willcocks. 2020. Robo-Apocalypse cancelled? Reframing the automation and future of work debate. Journal of Information Technology, 35, 4 (2020), 286-302. https://doi.org/10.1177/0268396220925830

[102]

Michael Williams, Hassen Gharbi, Aybike Ulusan, Ozlem Ergun, Zhu Xiaofeng, Shiyu Zhang and Casper Harteveld. 2016. Toward Human in the Loop Optimization Through Game-Based Experiments. In Proceedings of the Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. https://doi.org/10.1145/2968120.2987733

Digital Library

[103]

World Economic Forum. 2022. Without universal AI literacy, AI will fail us. Retrieved August 10, 2022 from https://www.weforum.org/agenda/2022/03/without-universal-ai-literacy-ai-will-fail-us/

[104]

Y. Xie, I.P. Bodala, D. C. Ong, D. Hsu and H. Soh. 2019. Robot Capability and Intention in Trust-Based Decisions Across Tasks. In Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[105]

Su-Fang Yeh, Meng-Hsin Wu, Tze-Yu Chen, Yen-Chun Lin, XiJing Chang, You-Hsuan Chiang and Yung-Ju Chang. 2022. How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3501941

Digital Library

[106]

Sangseok You, Cathy Liu Yang and Xitong Li. 2022. Algorithmic versus Human Advice: Does Presenting Prediction Performance Matter for Algorithm Appreciation? Journal of Management Information Systems, 39, 2 (2022), 336-365. https://doi.org/10.1080/07421222.2022.2063553

[107]

Gary Yukl and Ping P. Fu. 1999. Determinants of Delegation and Consultation by Managers. Journal of Organizational Behavior, 20, 2 (1999), 219-232. https://www.jstor.org/stable/3100422

[108]

Jichen Zhu, Jennifer Villareale, Nithesh Javvaji, Sebastian Risi, Mathias Löwe, Rush Weigelt and Casper Harteveld. 2021. Player-AI Interaction: What Neural Network Games Reveal About AI as Play. In Proceedings of the Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445307

Digital Library

Cited By

Ding HFan ZGuehring IGupta GHa WHuan JLiu LOmidvar-Tehrani BWang SZhou HBaeza-Yates RBonchi F(2024)Reasoning and Planning with Large Language Models in Code DevelopmentProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671452(6480-6490)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671452
Omidvar Tehrani BM IAnubhai A(2024)Evaluating Human-AI Partnership for LLM-based Code MigrationExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650896(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650896
Salikutluk VSchöpper JHerbert FScheuermann KFrodl EBalfanz DJäkel FKoert D(2024)An Evaluation of Situational Autonomy for Human-AI Collaboration in a Shared Workspace SettingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642564(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642564
Show More Cited By

Index Terms

AI Knowledge: Improving AI Delegation through Human Enablement
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Collaborative and social computing
  2. Human computer interaction (HCI)

Recommendations

Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
IUI '23: Proceedings of the 28th International Conference on Intelligent User Interfaces

Recent work has proposed artificial intelligence (AI) models that can learn to decide whether to make a prediction for an instance of a task or to delegate it to a human by considering both parties’ capabilities. In simulations with synthetically ...
ActiveAI: The Effectiveness of an Interactive Tutoring System in Developing K-12 AI Literacy
Technology Enhanced Learning for Inclusive and Equitable Quality Education
Abstract
As we witness groundbreaking advancements in Artificial Intelligence (AI), it is clear that the next generation must be equipped with AI literacy: the skill to interact, evaluate, and collaborate with AI systems. This study introduces ActiveAI, a ...
AI Literacy: Finding Common Threads between Education, Design, Policy, and Explainability
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Fostering public AI literacy has been a growing area of interest at CHI for several years, and a substantial community is forming around issues such as teaching children how to build and program AI systems, designing learning experiences to broaden ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

14911 pages

ISBN:9781450394215

DOI:10.1145/3544548

Editors:
Albrecht Schmidt
LMU Munich, Germany60028717
,
Kaisa Väänänen
Tampere University, Finland60011170
,
Tesh Goyal
Google Research, USA60006191
,
Per Ola Kristensson
University of Cambridge, UK60031101
,
Anicia Peters
University of Namibia, Namibia60072704
,
Stefanie Mueller
Massachusetts Institute of Technology, USA60022195
,
Julie R. Williamson
University of Glasgow, UK60001490
,
Max L. Wilson
University of Nottingham, UK60015138

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
2,259
Total Downloads

Downloads (Last 12 months)1,607
Downloads (Last 6 weeks)238

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ding HFan ZGuehring IGupta GHa WHuan JLiu LOmidvar-Tehrani BWang SZhou HBaeza-Yates RBonchi F(2024)Reasoning and Planning with Large Language Models in Code DevelopmentProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671452(6480-6490)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671452
Omidvar Tehrani BM IAnubhai A(2024)Evaluating Human-AI Partnership for LLM-based Code MigrationExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650896(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650896
Salikutluk VSchöpper JHerbert FScheuermann KFrodl EBalfanz DJäkel FKoert D(2024)An Evaluation of Situational Autonomy for Human-AI Collaboration in a Shared Workspace SettingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642564(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642564
Erlei ASharma AGadiraju U(2024)Understanding Choice Independence and Error Types in Human-AI CollaborationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641946(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641946
Ma SChen Z(2024)The Development and Validation of the Artificial Intelligence Literacy Scale for Chinese College Students (AILS-CCS)IEEE Access10.1109/ACCESS.2024.346837812(146419-146429)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3468378
Li HKim S(2024)Developing AI literacy in HRD: competencies, approaches, and implicationsHuman Resource Development International10.1080/13678868.2024.233796227:3(345-366)Online publication date: 5-Apr-2024
https://doi.org/10.1080/13678868.2024.2337962
Pinski MHofmann TBenlian A(2024)AI Literacy for the top management: An upper echelons perspective on corporate AI orientation and implementation abilityElectronic Markets10.1007/s12525-024-00707-134:1Online publication date: 1-Apr-2024
https://doi.org/10.1007/s12525-024-00707-1
Jin HUchida M(2024)A Three-Year Analysis of Human Preferences in Delegating Tasks to AIArtificial Intelligence in HCI10.1007/978-3-031-60606-9_4(48-66)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60606-9_4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents