Debugging for Inclusivity in Online CS Courseware: Does it Work?
Authors: 
Amreeta Chatterjee, Rudrajit Choudhuri, Mrinmoy Sarkar, Soumiki Chattopadhyay, Dylan Liu, Samarendra Hedaoo, Margaret Burnett, Anita SarmaAuthors Info & Claims
Pages 419 - 433
Abstract
Online computer science (CS) courses have broadened access to CS education, yet inclusivity barriers persist for minoritized groups in these courses. One problem that recent research has shown is that often inclusivity biases (“inclusivity bugs”) lurk within the course materials themselves, disproportionately disadvantaging minoritized students. To address this issue, we investigated how a faculty member can use AID—an Automated Inclusivity Detector tool—to remove such inclusivity bugs from a large online CS1 (Intro CS) course and what is the impact of the resulting inclusivity fixes on the students’ experiences. To enable this evaluation, we first needed to (Bugs): investigate inclusivity challenges students face in 5 online CS courses; (Build): build decision rules to capture these challenges in courseware (“inclusivity bugs”) and implement them in the AID tool; (Faculty): investigate how the faculty member followed up on the inclusivity bugs that AID reported; and (Students): investigate how the faculty member’s changes impacted students’ experiences via a before-vs-after qualitative study with CS students. Our results from (Bugs) revealed 39 inclusivity challenges spanning courseware components from the syllabus to assignments. After implementing the rules in the tool (Build), our results from (Faculty) revealed how the faculty member treated AID more as a “peer” than an authority in deciding whether and how to fix the bugs. Finally, the study results with (Students) revealed that students found the after-fix courseware more approachable - feeling less overwhelmed and more in control in contrast to the before-fix version where they constantly felt overwhelmed, often seeking external assistance to understand course content.
References
[1]
Steven Abney. 1997. Part-of-speech tagging and partial parsing. In Corpus-based methods in language and speech processing. Springer, 118–136.
[2]
I Elaine Allen and Jeff Seaman. 2010. Learning on demand: Online education in the United States, 2009.Babson Survey Research Group, United States of America.
[3]
Manon Arcand and Jacques Nantel. 2012. Uncovering the nature of information processing of men and women online: The comparison of two models using the think-aloud method. Journal of Theoretical and Applied Electronic Commerce Research 7, 2 (2012), 106–120.
[4]
Rachel Baker, Thomas Dee, Brent Evan, and June John. 2018. Bias in Online Classes: Evidence from a Field Experiment. Technical Report. Stanford Center for Education Policy Analysis. https://cepa.stanford.edu/sites/default/files/wp18-03-201803.pdf
[5]
William H. Bares, Bill Manaris, Renée McCauley, and Christine Moore. 2019. Achieving Gender Balance through Creative Expression. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY, USA, 293–299. https://doi.org/10.1145/3287324.3287435
[6]
Lecia J Barker, Charlie McDowell, and Kimberly Kalahar. 2009. Exploring factors that influence computer science introductory course students to persist in the major. ACM Sigcse Bulletin 41, 1 (2009), 153–157.
[7]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
[8]
Lori Breslow, David E Pritchard, Jennifer DeBoer, Glenda S Stump, Andrew D Ho, and Daniel T Seaton. 2013. Studying learning in the worldwide classroom research into edX’s first MOOC.Research & Practice in Assessment 8 (2013), 13–25.
[9]
Michael Buckley, Helene Kershner, Kris Schindler, Carl Alphonce, and Jennifer Braswell. 2004. Benefits of using socially-relevant projects in computer science and engineering education. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education. 482–486.
[10]
Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016. GenderMag: A method for evaluating software’s gender inclusiveness. Interacting with Computers 28, 6 (2016), 760–787.
[11]
Shuo Chang, Vikas Kumar, Eric Gilbert, and Loren G Terveen. 2014. Specialization, homophily, and gender in a social curation site: Findings from Pinterest. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 674–686.
[12]
Gary Charness and Uri Gneezy. 2012. Strong Evidence for Gender Differences in Risk Taking. Journal of Economic Behavior & Organization - J ECON BEHAV ORGAN 83, 1 (06 2012), 50–58. https://doi.org/10.1016/j.jebo.2011.06.007
[13]
Amreeta Chatterjee, Mariam Guizani, Catherine Stevens, Jillian Emard, Mary Evelyn May, Margaret Burnett, Iftekhar Ahmed, and Anita Sarma. 2021. AID: An automated detector for gender-inclusivity bugs in OSS project pages. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1423–1435.
[14]
Amreeta Chatterjee, Lara Letaw, Rosalinda Garcia, Doshna Umma Reddy, Rudrajit Choudhuri, Sabyatha Sathish Kumar, Patricia Morreale, Anita Sarma, and Margaret Burnett. 2022. Inclusivity bugs in online courseware: A field study. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 356–372.
[15]
Ingrid Maria Christensen, Melissa Høegh Marcher, Paweł Grabarczyk, Therese Graversen, and Claus Brabrand. 2021. Computing Educational Activities Involving People Rather Than Things Appeal More to Women (Recruitment Perspective). In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 127–144. https://doi.org/10.1145/3446871.3469758
[16]
Online Learning Consortium. 2022. Quality Framework. OLC. https://onlinelearningconsortium.org/about/quality-framework-five-pillars/(accessed Mar. 2022).
[17]
Alexiei Dingli and Justin Mifsud. 2011. Useful: A framework to mainstream web site usability through automated evaluation. (2011).
[18]
Bob Dougherty and Alex Wade. 2008. Vischeck. Vischeck.com. http://www.vischeck.com/(accessed Mar, 2022).
[19]
Rodrigo Duran, Lassi Haaranen, and Arto Hellas. 2020. Gender Differences in Introductory Programming: Comparing MOOCs and Local Courses. Association for Computing Machinery, New York, NY, USA, 692–698. https://doi.org/10.1145/3328778.3366852
[20]
Brianna Dym, Namita Pasupuleti, Cole Rockwood, and Casey Fiesler. 2021. “You don’t do your hobby as a job”: Stereotypes of Computational Labor and their Implications for CS Education. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 823–829.
[21]
Holly Fiock. 2020. Designing a community of inquiry in online courses. The International Review of Research in Open and Distributed Learning 21, 1 (2020), 135–153.
[22]
Denae Ford, Justin Smith, Philip J Guo, and Chris Parnin. 2016. Paradise unplugged: Identifying barriers for female participation on stack overflow. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 846–857.
[23]
Anonymized ForReview. 2024. ICER 2024 submission materials. (3 2024). https://doi.org/10.6084/m9.figshare.25511869.v2
[24]
Hana Frluckaj, Laura Dabbish, David Gray Widder, Huilian Sophie Qiu, and James D Herbsleb. 2022. Gender and participation in open source software development. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–31.
[25]
Rosalinda Garcia, Patricia Morreale, Lara Letaw, Amreeta Chatterjee, Pankati Patel, Sarah Yang, Isaac Tijerina Escobar, Geraldine Jimena Noa, and Margaret Burnett. 2023. “Regular” CS × Inclusive Design= Smarter Students and Greater Diversity. ACM Transactions on Computing Education 23, 3 (2023), 1–35.
[26]
Rosalinda Garcia, Patricia Morreale, Gail Verdi, Heather Garcia, Jimena Noa Guevara, Spencer Madsen, Maria Jesus Alzugaray-Orellana, Elizabeth Li, and Margaret Burnett. 2024. The Matchmaker Inclusive Design Curriculum: A Faculty-Enabling Curriculum to Teach Inclusive Design Throughout Undergraduate CS. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems Proceedings (CHI’24).
[27]
D Randy Garrison, Terry Anderson, and Walter Archer. 1999. Critical inquiry in a text-based environment: Computer conferencing in higher education. The internet and higher education 2, 2-3 (1999), 87–105.
[28]
Philip J Guo and Katharina Reinecke. 2014. Demographic differences in how students navigate through MOOCs. In Proceedings of the first ACM conference on Learning@ scale conference. 21–30.
[29]
Md Montaser Hamid, Amreeta Chatterjee, Mariam Guizani, Andrew Anderson, Fatima Moussaoui, Sarah Yang, Isaac Escobar, Anita Sarma, and Margaret Burnett. 2023. How to measure diversity actionably in technology. Equity, Diversity, and Inclusion in Software Engineering: Best Practices and Insights (2023).
[30]
Karen Hamrick. 2022. Women, Minorities, and Persons with Disabilities in Science and Engineering. NSF National Center for Science and Engineering Statistics (NCSES). https://ncses.nsf.gov/pubs/nsf19304/digest/field-of-degree-women#computer-sciences(accessed Mar. 2022).
[31]
Hui-Ching Kayla Hsu and Nasir Memon. 2021. Crossing the Bridge to STEM: Retaining Women Students in an Online CS Conversion Program. ACM Transactions on Computing Education (TOCE) 21, 2 (2021), 1–16.
[32]
Melody Y Ivory. 2000. Web TANGO: towards automated comparison of information-centric web site designs. In CHI’00 extended abstracts on Human factors in computing systems. 329–330.
[33]
Leonard R Kasday. 2000. A tool to evaluate universal Web accessibility. In Proceedings on the 2000 conference on Universal Usability. 161–162.
[34]
Prem Nawaz Khan, Cathy O’Connor, and Srinivasu Chakravarthula. 2021. Automated Accessibility Testing Tool (AATT). AATT. https://github.com/paypal/AATT(accessed Mar, 2022).
[35]
René Kizilcec and Andrew Saltarelli. 2019. Psychologically Inclusive Design: Cues Impact Women’s Participation in STEM Education. In Proceedings of the 2019 CHI Conference on human factors in computing systems(CHI ’19). ACM, 1–10.
[36]
René F. Kizilcec and Sherif Halawa. 2015. Attrition and Achievement Gaps in Online Learning. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15). Association for Computing Machinery, New York, NY, USA, 57–66. https://doi.org/10.1145/2724660.2724680
[37]
Maria Klawe. 2013. Increasing female participation in computing: The Harvey Mudd College story. Computer 46, 3 (2013), 56–58.
[38]
Sophia Krause-Levy, William G. Griswold, Leo Porter, and Christine Alvarado. 2021. The Relationship Between Sense of Belonging and Student Outcomes in CS1 and Beyond(ICER 2021). Association for Computing Machinery, New York, NY, USA, 29–41. https://doi.org/10.1145/3446871.3469748
[39]
Sophia Krause-Levy, Mia Minnes, Christine Alvarado, and Leo Porter. 2021. Experience report: Designing massive open online computer science courses for inclusion. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1. 95–101.
[40]
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174.
[41]
Lara Letaw, Rosalinda Garcia, Heather Garcia, Christopher Perdriau, and Margaret Burnett. 2021. Changing the Online Climate via the Online Students: Effects of Three Curricular Interventions on Online CS Students’ Inclusivity. In Proceedings of the 17th ACM Conference on International Computing Education Research. ACM, Online, 42–59.
[42]
Patrick R Lowenthal and Charles B Hodges. 2015. In search of quality: Using quality matters to analyze the quality of massive, open, online courses (MOOCs). International Review of Research in Open and Distributed Learning 16, 5 (2015), 83–101.
[43]
Jennifer Mankoff. 2006. Practical service learning issues in HCI. In CHI’06 Extended Abstracts on Human Factors in Computing Systems. 201–206.
[44]
Melissa Høegh Marcher, Ingrid Maria Christensen, Paweł Grabarczyk, Therese Graversen, and Claus Brabrand. 2021. Computing Educational Activities Involving People Rather Than Things Appeal More to Women (CS1 Appeal Perspective). In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 145–156. https://doi.org/10.1145/3446871.3469761
[45]
Quality Matters. 2022. Enhancing Inclusiveness within the Quality Matters Framework. Quality Matters. https://www.qualitymatters.org/qa-resources/resource-center/conference-presentations/enhancing-inclusiveness-within-quality(accessed Mar. 2022).
[46]
Quality Matters. 2022. Quality Matters. https://www.qualitymatters.org/ (accessed Mar. 2022).
[47]
Christopher Mendez, Zoe Steine Hanson, Alannah Oleson, Amber Horvath, Charles Hill, Claudia Hilderbrand, Anita Sarma, and Margaret Burnett. 2018. Semi-Automating (or not) a Socio-Technical Method for Socio-Technical Systems. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 23–32.
[48]
Christopher Mendez, Hema Susmita Pedala, Zoe Steine-Hanson, Claudia Hilderbrand, Amber Horvath, Charles Hill, Logan Simpson, Nupoor Patil, Anita Sarma, and Margaret Burnett. 2017. Open source barriers to entry, revisited: A tools perspective. (2017).
[49]
NCES. 2021. Digest of Education Statistics, Table 311.15. National Center for Education Statistics. https://nces.ed.gov/programs/digest/d20/tables/dt20_311.15.asp(accessed Mar. 2022).
[50]
Alannah Oleson, Christopher Mendez, Zoe Steine-Hanson, Claudia Hilderbrand, Christopher Perdriau, Margaret Burnett, and Amy J. Ko. 2018. Pedagogical Content Knowledge for Teaching Inclusive Design. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). Association for Computing Machinery, New York, NY, USA, 69–77. https://doi.org/10.1145/3230977.3230998
[51]
Shailendra Palvia, Prageet Aeron, Parul Gupta, Diptiranjan Mahapatra, Ratri Parida, Rebecca Rosner, and Sumita Sindhi. 2018. Online education: Worldwide status, challenges, trends, and implications., 233–241 pages.
[52]
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193 (2021).
[53]
Krystle Phirangee and Alesia Malec. 2017. Othering in online learning: An examination of social presence, identity, and sense of community. Distance Education 38, 2 (2017), 160–172.
[54]
Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643.
[55]
Debashish Pradhan, Tripti Rajput, Aravind Jembu Rajkumar, Jonathan Lazar, Rajiv Jain, Vlad I. Morariu, and Varun Manjunatha. 2022. Development and Evaluation of a Tool for Assisting Content Creators in Making PDF Files More Accessible. ACM Trans. Access. Comput. 15, 1, Article 3 (mar 2022), 52 pages. https://doi.org/10.1145/3507661
[56]
James Pustejovsky and Branimir Boguraev. 1993. Lexical knowledge representation and natural language processing. Artificial Intelligence 63, 1-2 (1993), 193–223.
[57]
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic keyword extraction from individual documents. Text mining: applications and theory (2010), 1–20.
[58]
Ali Akbar Septiandri, Marios Constantinides, Mohammad Tahaei, and Daniele Quercia. 2023. WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 160–171.
[59]
Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L Glassman. 2023. Where to hide a stolen elephant: Leaps in creative writing with multimodal machine intelligence. ACM Transactions on Computer-Human Interaction 30, 5 (2023), 1–57.
[60]
Christine K Sorensen and Danilo M Baylen. 2009. Learning online. DISTANCE LEARNING EDITORS AND EDITORIAL ADVISORY BOARD 7 (2009).
[61]
Steven E Stemler. 2004. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research, and Evaluation 9, 1 (2004), 4.
[62]
Simone Stumpf, Anicia Peters, Shaowen Bardzell, Margaret Burnett, Daniela Busse, Jessica Cauchard, and Elizabeth Churchill. 2020. Gender-inclusive HCI research and design: A conceptual review. Foundations and Trends in Human–Computer Interaction 13, 1 (2020), 1–69.
[63]
Adrian Thinnyun, Ryan Lenfant, Raymond Pettit, and John R Hott. 2021. Gender and Engagement in CS Courses on Piazza. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 438–444.
[64]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
[65]
Bogdan Vasilescu, Andrea Capiluppi, and Alexander Serebrenik. 2014. Gender, representation and online participation: A quantitative study. Interacting with Computers 26, 5 (2014), 488–511.
[66]
Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe Steine-Hanson, and Margaret Burnett. 2019. From Gender Biases to Gender-Inclusive Design: An Empirical Investigation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300283
[67]
Jayce R Warner, Carol L Fletcher, Nicole D Martin, and Stephanie N Baker. 2021. Applying the CAPE framework to measure equity and inform policy in computer science education. Policy Futures in Education (2021), 14782103221074467.
[68]
Linda L Werner, Brian Hanks, and Charlie McDowell. 2004. Pair-programming helps female computer science students. Journal on Educational Resources in Computing (JERIC) 4, 1 (2004), 4–es.
[69]
Julia Yates and Anke C Plagnol. 2022. Female computer science students: A qualitative exploration of women’s experiences studying computer science at university in the UK. Education and Information Technologies 27, 3 (2022), 3079–3105.
[70]
Kimberly Michelle Ying, Lydia G Pezzullo, Mohona Ahmed, Kassandra Crompton, Jeremiah Blanchard, and Kristy Elizabeth Boyer. 2019. In their own words: Gender differences in student perceptions of pair programming. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1053–1059.
Index Terms
- Debugging for Inclusivity in Online CS Courseware: Does it Work?
Recommendations
Inclusivity Bugs in Online Courseware: A Field Study
ICER '22: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1Motivation: Although asynchronous online CS courses have enabled more diverse populations to access CS higher education, research shows that online CS-ed is far from inclusive, with women and other underrepresented groups continuing to face inclusion ...
Changing the Online Climate via the Online Students: Effects of Three Curricular Interventions on Online CS Students’ Inclusivity
ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education ResearchMotivation: Although CS Education researchers and practitioners have found ways to improve CS classroom inclusivity, few researchers have considered inclusivity of online CS education. We are interested in two such improvements in online CS education—...
Implementing Inclusive Software Design in the CS Curriculum
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2Methods for teaching computer science (CS) have changed very little in the past decades. As a result, students graduating from CS/IT programs, educated in the same manner year after year, develop software with the same prior limitations and exclusionary ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2024
539 pages
ISBN:9798400704758
DOI:10.1145/3632620
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- USDA-NIFA/NSF
- NSF
Conference
ICER 2024
Sponsor:
ICER 2024: ACM Conference on International Computing Education Research
August 13 - 15, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 189 of 803 submissions, 24%
Upcoming Conference
ICER 2025
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 328Total Downloads
- Downloads (Last 12 months)328
- Downloads (Last 6 weeks)59
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in