Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3491102.3517439acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support

Published: 28 April 2022 Publication History

Abstract

AI-based decision support tools (ADS) are increasingly used to augment human decision-making in high-stakes, social contexts. As public sector agencies begin to adopt ADS, it is critical that we understand workers’ experiences with these systems in practice. In this paper, we present findings from a series of interviews and contextual inquiries at a child welfare agency, to understand how they currently make AI-assisted child maltreatment screening decisions. Overall, we observe how workers’ reliance upon the ADS is guided by (1) their knowledge of rich, contextual information beyond what the AI model captures, (2) their beliefs about the ADS’s capabilities and limitations relative to their own, (3) organizational pressures and incentives around the use of the ADS, and (4) awareness of misalignments between algorithmic predictions and their own decision-making objectives. Drawing upon these findings, we discuss design implications towards supporting more effective human-AI decision-making.

Supplemental Material

ZIP File
Supplemental Materials

References

[1]
Ali Alkhatib. 2021. To live in their utopia: Why algorithmic systems create absurd outcomes. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–9.
[2]
Ali Alkhatib and Michael Bernstein. 2019. Street-level algorithms: A theory at the gaps between policy and decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[3]
Asbjørn Ammitzbøll Flügge, Thomas Hildebrandt, and Naja Holten Møller. 2021. Street-Level algorithms and AI in bureaucratic decision-making: A caseworker perspective. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–23.
[4]
Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7, 1(2019), 19. www.aaai.org
[5]
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16.
[6]
Eric PS Baumer and M Six Silberman. 2011. When the implication is not to design (technology). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2271–2274.
[7]
Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42.
[8]
Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[9]
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–21.
[10]
Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019. ” Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making. Proceedings of the ACM on Human-computer Interaction 3, CSCW(2019), 1–24.
[11]
Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021. Soliciting stakeholders’ fairness notions in child maltreatment predictive systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.
[12]
Marc Cherna. [n.d.]. We will use all resources to keep children safe. Pittsburgh Post-Gazette([n. d.]). https://www.post-gazette.com/opinion/letters/2018/03/23/We-will-use-all-resources-to-keep-children-safe/stories/201803230094 Online; accessed 8-September-2021.
[13]
Alexandra Chouldechova, Emily Putnam-Hornstein, Suzanne Dworak-Peck, Diana Benavides-Prado, Oleksandr Fialko, Rhema Vaithianathan, Sorelle A Friedler, and Christo Wilson. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. Proceedings of Machine Learning Research 81 (2018), 1–15. http://proceedings.mlr.press/v81/chouldechova18a.html
[14]
Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. 2021. Leveraging expert consistency to improve algorithmic decision support. arXiv (2021), 1–33. arxiv:2101.09648http://arxiv.org/abs/2101.09648
[15]
Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020. A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. arXiv (2020), 1–12.
[16]
Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.
[17]
Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin’s Press.
[18]
Ben Green. 2021. The flaws of policies requiring human oversight of government algorithms. Available at SSRN (2021).
[19]
Ben Green and Yiling Chen. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019). https://doi.org/10.1145/3359152
[20]
Ron Haskins. 2020. Child welfare financing: What do we fund, how, and what could be improved?The ANNALS of the American Academy of Political and Social Science 692, 1(2020), 50–67.
[21]
Kenneth Holstein and Vincent Aleven. 2021. Designing for human-AI complementarity in K-12 education. arXiv preprint arXiv:2104.01266(2021).
[22]
Kenneth Holstein, Vincent Aleven, and Nikol Rummel. 2020. A conceptual framework for human–AI hybrid adaptivity in education. In International Conference on Artificial Intelligence in Education. Springer, 240–254.
[23]
Kenneth Holstein, Erik Harpstead, Rebecca Gulotta, and Jodi Forlizzi. 2020. Replay Enactments: Exploring possible futures through historical data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1607–1618.
[24]
Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2018. Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In International Conference on Artificial Intelligence in Education. 154–168.
[25]
Kenneth Holstein, Bruce M Mclaren, and Vincent Aleven. 2019. Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics 6, 2 (2019), 27–52.
[26]
Naja Holten Møller, Irina Shklovski, and Thomas T. Hildebrandt. 2020. Shifting concepts of value: Designing algorithmic decision-support systems for public services. NordiCHI (2020), 1–12. https://doi.org/10.1145/3419249.3420149
[27]
Dan Hurley. 2018. Can an algorithm tell when kids are in danger. New York Times 2(2018).
[28]
D. Jackson and G. Marx. 2017. Data mining program designed to predict child abuse proves unreliable, DCFS says.http://www.jstor.org/stable/10.7758/9781610447713
[29]
Will Johnson. 2004. Effectiveness of California’s child welfare Structured Decision-Making (SDM) model: A Prospective study of the validity of the California family risk assessment.
[30]
Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. 2021. Noise: A flaw in human judgment. Little, Brown.
[31]
Ece Kamar. 2016. Directions in hybrid intelligence: Complementing AI systems with human intelligence. International Joint Conference on Artificial Intelligence (IJCAI) (2016), 4070–4073.
[32]
Emily Keddell. 2019. Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice. Social Sciences (2019).
[33]
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237–293.
[34]
Janin Koch and Antti Oulasvirta. 2018. Group cognition and collaborative AI. In Human and Machine Learning. Springer, 293–312.
[35]
Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1–10.
[36]
Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. arXiv preprint arXiv:2112.11471(2021).
[37]
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences 40 (2017).
[38]
John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
[39]
Karen Levy, Kyla E Chasalow, and Sarah Riley. 2021. Algorithms and Decision-Making in the Public Sector. Annual Review of Law and Social Science 17 (2021), 1–38.
[40]
Michael Lipsky. 1980. Street Level Bureaucracy. Russell Sage Foundation. http://www.jstor.org/stable/10.7758/9781610447713
[41]
Emma Lurie and Deirdre K Mulligan. 2020. Crowdworkers are not judges: Rethinking crowdsourced vignette studies as a risk assessment evaluation technique. Proceedings of the Workshop on Fair and Responsible AI at CHI 2020 (2020).
[42]
Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–23.
[43]
Hussein Mozannar, Arvind Satyanarayan, and David Sontag. 2021. Teaching Humans When To Defer to a Classifier via Examplars. arXiv preprint arXiv:2111.11297(2021).
[44]
Toni Naccarato. 2010. Child welfare informatics: A proposed subspecialty for social work. Children and Youth Services Review 32, 12 (2010), 1729–1734.
[45]
Laura Nissen. 2020. Social work and the future in a post-Covid 19 world: A foresight lens and a call to action for the profession. Journal of Technology in Human Services 38, 4 (2020), 309–330.
[46]
Allegheny County Department of Human Services. [n.d.]. The Allegheny Family Screening Tool. https://www.alleghenycounty.us/Human-Services/News-Events/Accomplishments/Allegheny-Family-Screening-Tool.aspx. Online; accessed 7-September-2021.
[47]
Allegheny County Department of Human Services. [n.d.]. Allegheny Family Screening Tool, Frequently-Asked Questions | Updated August 2018. https://www.alleghenycountyanalytics.us/wp-content/uploads/2018/10/17-ACDHS-11_AFST_102518.pdf. Online; accessed 8-September-2021.
[48]
Bhavik N Patel, Louis Rosenberg, Gregg Willcox, David Baltaxe, Mimi Lyons, Jeremy Irvin, Pranav Rajpurkar, Timothy Amrhein, Rajan Gupta, Safwan Halabi, Curtis Langlotz, Edward Lo, Joseph Mammarappallil, A J Mariano, Geoffrey Riley, Jayne Seekins, Luyao Shen, Evan Zucker, and Matthew P Lungren. 2019. Human-machine partnership with artificial intelligence for chest radiograph diagnosis. npj Digital Medicine(2019). https://doi.org/10.1038/s41746-019-0189-7
[49]
Jennifer Pierre, Roderic Crooks, Morgan Currie, Britt Paris, and Irene Pasquetto. 2021. Getting Ourselves Together: Data-centered participatory design research & epistemic burden. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–11.
[50]
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and measuring model interpretability. (2021). arxiv:1802.07810
[51]
Anjana Samant, Aaron Horowitz, Kath Xu, and Sophie Beiers. 2021. Family surveillance by algorithm: The rapidly spreading tools few have heard of. American Civil Liberties Union (ACLU)(2021). https://www.aclu.org/sites/default/files/field_document/2021.09.28a_family_surveillance_by_algorithm.pdf
[52]
Devansh Saxena, Karla Badillo-Urquiola, Pamela Wisniewski, and Shion Guha. 2021. A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child welfare. arXiv 5, October (2021). arxiv:arXiv:2107.03487v2
[53]
Devansh Saxena, Karla Badillo-Urquiola, Pamela J Wisniewski, and Shion Guha. 2020. A human-centered review of algorithms used within the US child welfare system. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.
[54]
China Scherz. 2011. Protecting children, preserving families: Moral conflict and actuarial science in a problem of contemporary governance. PoLAR: Political and Legal Anthropology Review 34, 1 (2011), 33–50.
[55]
Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021. Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. arXiv preprint arXiv:2105.02980(2021).
[56]
C. Estelle Smith, Bowen Yu, Anjali Srivastava, Aaron Halfaker, Loren Terveen, and Haiyi Zhu. 2020. Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems. Conference on Human Factors in Computing Systems - Proceedings (2020), 1–14. https://doi.org/10.1145/3313831.3376783
[57]
Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Towards a process model for co-creating AI experiences. arXiv preprint arXiv:2104.07595(2021).
[58]
Sarah Tan, Julius Adebayo, and Kori Inkpen. 2018. Investigating human + machine complementarity for recidivism predictions. arXiv (2018). arxiv:arXiv:1808.09123v2
[59]
Rhema Vaithianathan, Diana Benavides-Prado, Erin Dalton, Alex Chouldechova, and Emily Putnam-Hornstein. 2021. Using a machine learning tool to support high-stakes decisions in child protection. AI Magazine 42, 1 (2021), 53–60.
[60]
Rhema Vaithianathan, Emily Putnam-Hornstein, Nan Jiang, Parma Nand, and Tim Maloney. 2017. Developing predictive models to support child maltreatment hotline screening decisions: Allegheny County methodology and implementation. Center for Social data Analytics(2017).
[61]
Piet Van den Bossche, Wim Gijselaers, Mien Segers, Geert Woltjer, and Paul Kirschner. 2011. Team learning: Building shared mental models. Instructional Science 39, 3 (2011), 283–301.
[62]
Elmira van den Broek, Anastasia Sergeeva, and Marleen Huysman. 2020. Hiring algorithms: An ethnography of fairness in practice. 40th International Conference on Information Systems, ICIS 2019 (2020).
[63]
MW Van Someren, YF Barnard, and JAC Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress(1994).
[64]
Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–14.
[65]
James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viégas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics 26, 1(2019), 56–65.
[66]
Qian Yang, Aaron Steinfeld, and John Zimmerman. 2019. Unremarkable AI: Fitting intelligent decision support into critical, clinical decision-making processes. Conference on Human Factors in Computing Systems - Proceedings (2019). https://doi.org/10.1145/3290605.3300468 arxiv:1904.09612
[67]
Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F Antaki. 2016. Investigating the heart pump implant decision process: opportunities for decision support tools to help. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4477–4488.
[68]
Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018. Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human-Computer Interaction 2, CSCW(2018), 1–23.
[69]
Alexandra Zytek, Dongyu Liu, Rhema Vaithianathan, and Kalyan Veeramachaneni. 2021. Sibyl: Understanding and addressing the usability challenges of machine learning in high-stakes decision making. IEEE Transactions on Visualization and Computer Graphics (2021).

Cited By

View all
  • (2025)AI in Social Worker Recruitment and Training for Bullying InterventionSustainable Management Practices for Employee Retention and Recruitment10.4018/979-8-3693-8679-8.ch001(1-30)Online publication date: 28-Feb-2025
  • (2025)Artificial Intelligence Algorithms, Bias, and Innovation: Implications for Social WorkJournal of Evidence-Based Social Work10.1080/26408066.2025.2470903(1-23)Online publication date: 26-Feb-2025
  • (2025)Social Work Educators Innovating With Generative AI: An Exploratory StudyJournal of Social Work Education10.1080/10437797.2024.2411170(1-16)Online publication date: 3-Jan-2025
  • Show More Cited By

Index Terms

  1. Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support
            Index terms have been assigned to the content through auto-classification.

            Recommendations

            Comments

            Please enable JavaScript to view thecomments powered by Disqus.

            Information & Contributors

            Information

            Published In

            cover image ACM Conferences
            CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
            April 2022
            10459 pages
            ISBN:9781450391573
            DOI:10.1145/3491102
            This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

            Sponsors

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            Published: 28 April 2022

            Check for updates

            Badges

            • Honorable Mention

            Author Tags

            1. algorithm-assisted decision making
            2. child welfare
            3. contextual inquiry
            4. decision support

            Qualifiers

            • Research-article
            • Research
            • Refereed limited

            Funding Sources

            Conference

            CHI '22
            Sponsor:
            CHI '22: CHI Conference on Human Factors in Computing Systems
            April 29 - May 5, 2022
            LA, New Orleans, USA

            Acceptance Rates

            Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

            Upcoming Conference

            CHI 2025
            ACM CHI Conference on Human Factors in Computing Systems
            April 26 - May 1, 2025
            Yokohama , Japan

            Contributors

            Other Metrics

            Bibliometrics & Citations

            Bibliometrics

            Article Metrics

            • Downloads (Last 12 months)3,318
            • Downloads (Last 6 weeks)480
            Reflects downloads up to 03 Mar 2025

            Other Metrics

            Citations

            Cited By

            View all
            • (2025)AI in Social Worker Recruitment and Training for Bullying InterventionSustainable Management Practices for Employee Retention and Recruitment10.4018/979-8-3693-8679-8.ch001(1-30)Online publication date: 28-Feb-2025
            • (2025)Artificial Intelligence Algorithms, Bias, and Innovation: Implications for Social WorkJournal of Evidence-Based Social Work10.1080/26408066.2025.2470903(1-23)Online publication date: 26-Feb-2025
            • (2025)Social Work Educators Innovating With Generative AI: An Exploratory StudyJournal of Social Work Education10.1080/10437797.2024.2411170(1-16)Online publication date: 3-Jan-2025
            • (2024)Disability and AI(Artificial Intelligence): Topic modeling of newspaperJournal of Families and Better Life10.7466/JFBL.2024.42.3.11942:3(119-136)Online publication date: 30-Sep-2024
            • (2024)Comparative analysis of the legal frameworks of the Republic of Serbia and Republic of Albania: The application of Artificial Intelligence and child protectionXXI međunarodni naučni skup „Pravnički dani - Prof. dr Slavko Carić“, Odgovori pravne nauke na izazove savremenog društva - zbornik radova10.5937/PDSC24413D(413-425)Online publication date: 2024
            • (2024)From Technostress to TurnoverModern Trends and Future Innovation in Human Resource Management10.4018/979-8-3693-6402-4.ch004(105-136)Online publication date: 25-Sep-2024
            • (2024)Ethical considerations in research when building predictive risk modelling in child and family welfareJournal of Comparative Social Work10.31265/jcsw.v19i1.61919:1(102-126)Online publication date: 3-Oct-2024
            • (2024)Beyond Recommendations: From Backward to Forward AI Support of Pilots' Decision-Making ProcessProceedings of the ACM on Human-Computer Interaction10.1145/36870248:CSCW2(1-32)Online publication date: 8-Nov-2024
            • (2024)Studying Up Public Sector AI: How Networks of Power Relations Shape Agency Decisions Around AI Design and UseProceedings of the ACM on Human-Computer Interaction10.1145/36869898:CSCW2(1-24)Online publication date: 8-Nov-2024
            • (2024)Reconfiguring Data Relations: Institutional Dynamics around Data in Local GovernanceProceedings of the ACM on Human-Computer Interaction10.1145/36869598:CSCW2(1-28)Online publication date: 8-Nov-2024
            • Show More Cited By

            View Options

            View options

            PDF

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format.

            HTML Format

            Login options

            Figures

            Tables

            Media

            Share

            Share

            Share this Publication link

            Share on social media