Supporting Storytelling With Evidence in Holistic Review Processes: A Participatory Design Approach
Authors: 
Ronald A. Metoyer, 
Tee Chuanromanee, Gina M. Girgis, 
Qiyu Zhi, Eleanor C. KinyonAuthors Info & Claims
Article No.: 61, Pages 1 - 24
Abstract
Review processes involve complex and often subjective decision-making tasks in which individual reviewers must read and rate submissions, such as a college application, along many relevant dimensions and typically with a rubric in mind. A common part of the work is committee review, where individual reviewers meet to discuss the merits of a particular submission in order to recommend an accept or reject decision. Prior work indicates that visualization and sensemaking support may be beneficial in such processes where reviewers must present the "story" of the applicant under question. We conducted a series of participatory design workshops with reviewers in the domain of holistic college admissions to better understand the challenges and opportunities regarding storytelling. Based on these workshops, we contribute a characterization for how reviewers in this domain construct visual stories, we provide guidance for designing for evidence capture and storytelling, and we draw parallels and distinctions between this domain and other reviewing domains.
Supplementary Material
The supplementary file contains 1) seven stylized collages from workshop # 2, 2) mockups for the five different design ideas from workshop #3, and 3) a description of the snapshot tool that reviewers provided feedback on. All are contained in a single .pdf file.
- Download
- 530.04 KB
References
[1]
Mark S Ackerman, Juri Dachtera, Volkmar Pipek, and Volker Wulf. 2013. Sharing knowledge and expertise: The CSCW view of knowledge management. Computer Supported Cooperative Work (CSCW), Vol. 22, 4--6 (2013), 531--573.
[2]
Sergio Alonso, Enrique Herrera-Viedma, Francisco Javier Cabrerizo, Carlos Porcel, and Antonio Gabriel López-Herrera. 2007. Using visualization tools to guide consensus in group decision making. In International Workshop on Fuzzy Logic and Applications. Springer, 77--85.
[3]
Jeff Baker, Donald Jones, and Jim Burkman. 2009. Using visual representations of data to enhance sensemaking in data exploration tasks. Journal of the Association for Information Systems, Vol. 10, 7 (2009), 2.
[4]
Michael N Bastedo, Joseph E Howard, and Allyson Flaster. 2016. Holistic admissions after affirmative action: Does "maximizing" the high school curriculum matter? Educational Evaluation and Policy Analysis, Vol. 38, 2 (2016), 389--409.
[5]
Alexander Boden, Frank Rosswog, Gunnar Stevens, and Volker Wulf. 2014. Articulation spaces: bridging the gap between formal and informal coordination. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1120--1130.
[6]
William G Bowen and Derek Bok. 1999. The shape of the river: Long-term consequences of considering race in college and university admissions .Princeton University Press Princeton, NJ.
[7]
Tone Bratteteig and Ina Wagner. 2016. What is a participatory design result?. In Proceedings of the 14th Participatory Design Conference: Full papers-Volume 1. ACM, 141--150.
[8]
Wayne J Camara and Amy Elizabeth Schmidt. 1999. Group Differences in Standardized Testing and Social Stratification. Report No. 99--5. College Entrance Examination Board (1999).
[9]
Maia B Cook and Harvey S Smallman. 2008. Human factors of the confirmation bias in intelligence analysis: Decision support from graphical evidence landscapes. Human Factors, Vol. 50, 5 (2008), 745--754.
[10]
Sarah E Drew, Rony E Duncan, and Susan M Sawyer. 2010. Visual storytelling: A beneficial but challenging method for health research with young people. Qualitative health research, Vol. 20, 12 (2010), 1677--1688.
[11]
Daniel Golden. 2007. The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates. Broadway Books.
[12]
James Hollan, Edwin Hutchins, and David Kirsh. 2000. Distributed cognition: toward a new foundation for human-computer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI), Vol. 7, 2 (2000), 174--196.
[13]
Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defining customer-centered systems .Elsevier.
[14]
M Shahriar Hossain, Patrick Butler, Arnold P Boedihardjo, and Naren Ramakrishnan. 2012. Storytelling in entity networks to support intelligence analysts. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1375--1383.
[15]
Joshua E Introne. 2009. Supporting group decisions by mediating deliberation to improve information pooling. In Proceedings of the ACM 2009 international conference on Supporting group work. ACM, 189--198.
[16]
Nipat Jongsawat and Wichian Premchaiswadi. 2009. Group awareness information in web-based group decision support system. In 2009 IEEE International Conference on Systems, Man and Cybernetics. IEEE, 370--375.
[17]
Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
[18]
Paul E Keel. 2007. EWall: A visual analytics environment for collaborative sense-making. Information Visualization, Vol. 6, 1 (2007), 48--63.
[19]
Benedikt Ley, Thomas Ludwig, Volkmar Pipek, Dave Randall, Christian Reuter, and Torben Wiedenhoefer. 2014. Information and expertise sharing in inter-organizational crisis management. Computer Supported Cooperative Work (CSCW), Vol. 23, 4--6 (2014), 347--387.
[20]
Narges Mahyar, Weichen Liu, Sijia Xiao, Jacob Browne, Ming Yang, and Steven P Dow. 2017. ConsesnsUs: Visualizing Points of Disagreement for Multi-Criteria Collaborative Decision Making. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 17--20.
[21]
Narges Mahyar and Melanie Tory. 2014. Supporting communication and coordination in collaborative sensemaking. IEEE transactions on visualization and computer graphics, Vol. 20, 12 (2014), 1633--1642.
[22]
Helena M Mentis, Paula M Bach, Blaine Hoffman, Mary Beth Rosson, and John M Carroll. 2009. Development of decision rationale in complex group decision making. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1341--1350.
[23]
Michael J Muller. 2009. Participatory design: the third space in HCI. In Human-computer interaction. CRC press, 181--202.
[24]
Iván Palomares and Luis Mart'inez. 2014. Low-dimensional visualization of experts' preferences in urgent group decision making under uncertainty. Procedia Computer Science, Vol. 29 (2014), 2090--2101.
[25]
Iván Palomares, Luis Mart'inez, and Francisco Herrera. 2014. MENTOR: A graphical monitoring tool of preferences evolution in large-scale group decision making. Knowledge-Based Systems, Vol. 58 (2014), 66--74.
[26]
Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2--4.
[27]
Katie Reilly. 2018 (accessed August 12, 2019). A Lawsuit by Asian-American Students Against Harvard Could End Affirmative Action as We Know It. https://time.com/5425147/harvard-affirmative-action-trial-asian-american-students/
[28]
Michael Ross and Fiore Sicoly. 1979. Egocentric biases in availability and attribution. Journal of personality and social psychology, Vol. 37, 3 (1979), 322.
[29]
Adam Rule, Aurélien Tabard, and Jim Hollan. 2017a. Using visual histories to reconstruct the mental context of suspended activities. Human-Computer Interaction, Vol. 32, 5--6 (2017), 511--558.
[30]
Adam Rule, Aurélien Tabard, and Jim Hollan. 2017b. Using visual histories to reconstruct the mental context of suspended activities. Human-Computer Interaction, Vol. 32, 5--6 (2017), 511--558.
[31]
Edward Segel and Jeffrey Heer. 2010. Narrative visualization: Telling stories with data. IEEE transactions on visualization and computer graphics, Vol. 16, 6 (2010), 1139--1148.
[32]
Ben Shneiderman. 1996. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings 1996 IEEE symposium on visual languages. IEEE, 336--343.
[33]
Jacques Steinberg. 2003. The gatekeepers: Inside the admissions process of a premier college .Penguin.
[34]
Anselm L Strauss. 1987. Qualitative analysis for social scientists .Cambridge university press.
[35]
Poorna Talkad Sukumar and Ronald Metoyer. 2018. A Visualization Approach to Addressing Reviewer Bias in Holistic College Admissions. In Cognitive Biases in Visualizations. Springer, 161--175.
[36]
Poorna Talkad Sukumar, Ronald Metoyer, and Shuai He. 2018. Making a Pecan Pie: Understanding and Supporting The Holistic Review Process in Admissions. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 169.
[37]
Jan M Van Bruggen, Henny PA Boshuizen, and Paul A Kirschner. 2003. A cognitive framework for cooperative problem solving with argument visualization. In Visualizing argumentation. Springer, 25--47.
[38]
Susan W van den Braak, H van Oostendorp, Henry Prakken, and Gerard AW Vreeswijk. 2008. A critical review of argument visualization tools: Do users become better reasoners?. In Workshop notes of the ECAI-06 workshop on computational models of natural argument. 67--75.
[39]
Douglas Walton. 2009. Argument visualization tools for corroborative evidence. In Proc. of the 2nd International Conference on Evidence Law and Forensic Science. 32--49.
[40]
Lu Xiao and Richelle L Witherspoon. 2015. Information sharing as story construction in group decision making. In Proceedings of the 78th ASIS&T Annual Meeting: Information Science with Impact: Research in and for the Community. American Society for Information Science, 63.
[41]
Lingxue Yang. 2018. UX design for memory supplementation to support problem-solving tasks in analytic applications. Ph.D. Dissertation.
[42]
Qiyu Zhi, Suwen Lin, Poorna Talkad Sukumar, and Ronald Metoyer. 2019. GameViews: Understanding and Supporting Data-driven Sports Storytelling. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 269.
Index Terms
- Supporting Storytelling With Evidence in Holistic Review Processes: A Participatory Design Approach
Recommendations
The role of design fiction in participatory design processes
NordiCHI '18: Proceedings of the 10th Nordic Conference on Human-Computer InteractionParticipatory design is in essence very malleable as any design technique could lend itself to it, as long as users and stakeholders are involved. Design fictions however, have more often been used as either a vehicle for critical designs, or as a sheer ...
Workshop on participatory design and the humanist landscape
UbiComp/ISWC'15 Adjunct: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable ComputersThis workshop (held in conjunction with the UBICOMP 2015: The 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and ISCW 2015: The 19th International Symposium on Wearable Computers) will bring together researchers and ...
The early shaping of participatory design at PDC
PDC '16: Proceedings of the 14th Participatory Design Conference: Short Papers, Interactive Exhibitions, Workshops - Volume 2For 25 years the Participatory Design Conference (PDC) has been concerned with the understanding and development of Participatory Design in theory and practice. The main contribution of this paper is an informed understanding of how the participatory ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 May 2020
Online AM: 07 May 2020
Published in PACMHCI Volume 4, Issue CSCW1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Science Foundation
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 350Total Downloads
- Downloads (Last 12 months)67
- Downloads (Last 6 weeks)2
Reflects downloads up to 30 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in