research-article

Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth

Authors:

Chun-Wei Chiang,

Saiph SavageAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 2, Issue CSCW

Article No.: 37, Pages 1 - 17

https://doi.org/10.1145/3274306

Published: 01 November 2018 Publication History

Abstract

Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.

Supplementary Material

suppl.mov (cscw037.mp4)

Supplemental video

Download
25.39 MB

References

[1]

Ali Alkhatib, Michael S Bernstein, and Margaret Levi. 2017. Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4599--4616.

Digital Library

[2]

Walter Ángel, Saiph Savage, and Nataly Moreno. 2015. Participatory Stoves: Designing Renewable Energy Technologies for the Rural Sector. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. ACM, 259--262.

Digital Library

[3]

Thomas A Angelo and K Patricia Cross. 1993. Classroom assessment techniques: A handbookfor college teachers. San Francisco: Jossey-Bas (1993).

[4]

Fabricio Balcazar, Bill L Hopkins, and Yolanda Suarez. 1985. A critical, objective review of performance feedback. Journal of Organizational Behavior Management, Vol. 7, 3--4 (1985), 65--89.

[5]

Martina Balestra, Coye Cheshire, Ofer Arazy, and Oded Nov. 2017. Investigating the Motivational Paths of Peer Production Newcomers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6381--6385.

Digital Library

[6]

Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal, Vol. 37 (2015), 543.

[7]

Jeffrey P Bigham, Raja Kushalnagar, Ting-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage. 2017a. On How Deaf People Might Use Speech to Control Devices. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 383--384.

Digital Library

[8]

Jeffrey P Bigham, Kristin Williams, Nila Banerjee, and John Zimmerman. 2017b. Scopist: Building a Skill Ladder into Crowd Transcription. (2017).

Digital Library

[9]

Stephen Billett. 2001. Learning through work: workplace affordances and individual engagement. Journal of Workplace Learning, Vol. 13, 5 (2001), 209--214.

[10]

David Boud, Ruth Cohen, and Jane Sampson. 1999. Peer learning and assessment. Assessment & Evaluation in Higher Education, Vol. 24, 4 (1999), 413--426.

[11]

Daren C Brabham. 2008. Crowdsourcing as a model for problem solving: An introduction and cases. Convergence, Vol. 14, 1 (2008), 75--90.

[12]

Scott R Braithwaite, Christophe Giraud-Carrier, Josh West, Michael D Barnes, and Carl Lee Hanson. 2016. Validating machine learning algorithms for Twitter data against established measures of suicidality. JMIR Mental Health, Vol. 3, 2 (2016).

[13]

Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper. 2016. Why would anybody do this?: Understanding older adults' motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246--2257.

Digital Library

[14]

Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design .Morgan Kaufmann.

Digital Library

[15]

Carrie J Cai, Shamsi T Iqbal, and Jaime Teevan. 2016. Chain reactions: The impact of order on microtask chains. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3143--3154.

Digital Library

[16]

Chris Callison-Burch. 2014. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing .

[17]

Patricia A Carlson and Frederick C Berry. 2003. Calibrated peer review/sup TM/and assessing learning outcomes. In Frontiers in Education Conference. IEEE, F3E1--6.

[18]

Susan Cartwright and Nicola Holmes. 2006. The meaning of work: The challenge of regaining employee engagement and reducing cynicism. Human Resource Management Review, Vol. 16, 2 (2006), 199--208.

[19]

Joel Chan, Steven Dang, and Steven P Dow. 2016. Comparing different sensemaking approaches for large-scale ideation. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2717--2728.

Digital Library

[20]

Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces .Springer New York, New York, NY, 377--392.

[21]

Chun-Wei Chiang, Eber Betanzos, and Saiph Savage. 2018. Exploring Blockchain for Trustful Collaborations between Immigrants and Governments. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems . ACM, LBW531.

Digital Library

[22]

Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A Hearst. 2015. Structuring interactions for large-scale synchronous peer learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1139--1152.

Digital Library

[23]

Xuefei Nancy Deng and KD Joshi. 2013. Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers' Perceptions of Crowdsourcing Career. (2013).

[24]

Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 238--247.

Digital Library

[25]

Martin Dittus and Licia Capra. 2017. Private Peer Feedback As Engagement Driver in Humanitarian Mapping. Proc. ACM Hum.-Comput. Interact., Vol. 1, CSCW, Article 40 (Dec. 2017), bibinfonumpages18 pages.

Digital Library

[26]

Mira Dontcheva, Robert R Morris, Joel R Brandt, and Elizabeth M Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3379--3388.

Digital Library

[27]

Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2623--2634.

Digital Library

[28]

Steven P Dow, Kate Heddleston, and Scott R Klemmer. 2009. The efficacy of prototyping under time constraints. In Proceedings of the seventh ACM conference on Creativity and cognition. ACM, 165--174.

Digital Library

[29]

David Durward, Ivo Blohm, and Jan Marco Leimeister. 2016. Crowd work. Business & Information Systems Engineering, Vol. 58, 4 (2016), 281--286.

[30]

Roberta L Duyff. 1999. The value of lifelong learning: key element in professional career development. Journal of the Academy of Nutrition and Dietetics, Vol. 99, 5 (1999), 538.

[31]

Darren Edge, Elly Searle, Kevin Chiu, Jing Zhao, and James A Landay. 2011. MicroMandarin: mobile language learning in context. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3169--3178.

Digital Library

[32]

Nancy Falchikov and Judy Goldfinch. 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research, Vol. 70, 3 (2000), 287--322.

[33]

Benjamin V Hanrahan, Jutta K Willamowski, Saiganesh Swaminathan, and David B Martin. 2015. TurkBench: Rendering the market for Turkers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1613--1616.

Digital Library

[34]

Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 449.

Digital Library

[35]

Brent Harrison and Mark O Riedl. 2016. Learning from stories: using crowdsourced narratives to train virtual agents. Proceedings of the Artificial Intelligence and Interactive Digital Entertainment (2016), 183--189.

[36]

John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research, Vol. 77, 1 (2007), 81--112.

[37]

Yurong He, Jennifer Preece, Carol Boston, Anne Bowser, Derek Hansen, and Jen Hammock. 2014. The Effects of Individualized Feedback on College Students' Contributions to Citizen Science. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW Companion '14). ACM, New York, NY, USA, 165--168.

Digital Library

[38]

Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology, Vol. 86, 6 (2001), 1232.

[39]

John B. Horrigan. 2016. Civic engagement in the digital age. Pew Internet & American Life Project (2016).

[40]

Philip N Howard, Saiph Savage, Claudia Flores Saviaga, Carlos Toxtli, and Andrés Monroy-Hemández. 2016. Social media, civic engagement, and the slacktivism hypothesis: Lessons from Mexico's “El Bronco”. Journal of International Affairs, Vol. 70, 1 (2016).

[41]

Ting-Hao Kenneth Huang, Walter S Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. " Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing .

[42]

Kazushi Ikeda and Michael S Bernstein. 2016. Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4111--4121.

Digital Library

[43]

Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620.

Digital Library

[44]

Lilly C Irani and M Silberman. 2016. Stories we tell about labor: Turkopticon and the trouble with design. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 4573--4586.

Digital Library

[45]

Nila Banerjee Jeffrey P. Bigham, Kristin Williams and John Zimmerman. 2017. Scopist: Building a Skill Ladder into Crowd Work. In Proceedings of the Web for All Conference (W4A '17). ACM, New York, NY, USA, 10.

Digital Library

[46]

Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk. In AMCIS, Vol. 11. 1--11.

[47]

Clare Kelliher and Deirdre Anderson. 2008. For better or for worse? An analysis of how flexible working practices influence employees' perceptions of job quality. The International Journal of Human Resource Management, Vol. 19, 3 (2008), 419--431.

[48]

Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318.

Digital Library

[49]

Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin, Vol. 119, 2 (1996), 254.

[50]

Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science, Vol. 36, 5 (2012), 757--798.

[51]

Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively Crowdsourcing Workflows with Turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1003--1012.

Digital Library

[52]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI), Vol. 20, 6 (2013), 33.

Digital Library

[53]

Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84.

Digital Library

[54]

Airi Lampinen, Victoria Bellotti, Coye Cheshire, and Mary Gray. 2016. CSCW and theSharing Economy: The Future of Platforms as Sites of Work Collaboration and Trust. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. ACM, 491--497.

Digital Library

[55]

Gary P Latham and Edwin A Locke. 1991. Self-regulation through goal setting. Organizational behavior and human decision processes, Vol. 50, 2 (1991), 212--247.

[56]

Carol A Lundberg. 2003. The influence of time-limitations, faculty, and peer relationships on adult student learning: A causal model. The Journal of Higher Education, Vol. 74, 6 (2003), 665--688.

[57]

David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 224--235.

Digital Library

[58]

Winter Mason and Duncan J Watts. 2010. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, Vol. 11, 2 (2010), 100--108.

Digital Library

[59]

Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 (2017).

[60]

Meredith Ringel Morris, Jeffrey P Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1867--1876.

Digital Library

[61]

Melissa M Nelson and Christian D Schunn. 2009. The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, Vol. 37, 4 (2009), 375--401.

[62]

Jeffrey Nichols and Jeon-Hyung Kang. 2012. Asking questions of targeted strangers on social networks. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 999--1002.

Digital Library

[63]

Jeffrey Nichols, Michelle Zhou, Huahai Yang, Jeon-Hyung Kang, and Xiao Hua Sun. 2013. Analyzing the quality of information solicited from targeted strangers on social media. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 967--976.

Digital Library

[64]

Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z Gajos. 2011. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 1--12.

Digital Library

[65]

Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207--215.

Digital Library

[66]

Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. (2010).

[67]

Melissa M Patchan and Christian D Schunn. 2015. Understanding the benefits of providing peer feedback: how students respond to peers' texts of varying quality. Instructional Science, Vol. 43, 5 (2015), 591--614.

[68]

Sunil Ramlall. 2004. A review of employee motivation theories and their implications for employee retention within organizations. Journal of American Academy of Business, Vol. 5, 1/2 (2004), 52--63.

[69]

Alex Rosenblat and Luke Stark. 2016. Algorithmic labor and information asymmetries: A case study of Uber's drivers. (2016).

[70]

Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2009. Who are the turkers? Worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).

[71]

Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, Vol. 55, 1 (2000), 68.

[72]

Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 813--822.

Digital Library

[73]

Jodi Schneider, Bluma S. Gelley, and Aaron Halfaker. 2014. Accept, Decline, Postpone: How Newcomer Productivity is Reduced in English Wikipedia by Pre-publication Review. In Proceedings of The International Symposium on Open Collaboration (OpenSym '14). ACM, New York, NY, USA, Article 26, bibinfonumpages10 pages.

Digital Library

[74]

M Silberman, Lilly Irani, and Joel Ross. 2010a. Ethics and tactics of professional crowdwork. XRDS: Crossroads, The ACM Magazine for Students, Vol. 17, 2 (2010), 39--43.

Digital Library

[75]

M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. 2010b. Sellers' Problems in Human Computation Markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 18--21.

Digital Library

[76]

Ryo Suzuki, Niloufar Salehi, Michelle S Lam, Juan C Marroquin, and Michael S Bernstein. 2016. Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2645--2656.

Digital Library

[77]

William Thies, Aishwarya Ratan, and James Davis. 2011. Paid crowdsourcing as a vehicle for global development. In CHI Workshop on Crowdsourcing and Human Computation .

[78]

David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology, Vol. 15, 1 (2013), 29.

[79]

Bonnie Urciuoli. 2008. Skills and selves in the new workplace. American Ethnologist, Vol. 35, 2 (2008), 211--228.

[80]

Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, and Michael S Bernstein. 2014. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3645--3654.

Digital Library

[81]

Niels van Doorn. 2017. Platform labor: on the gendered and racialized exploitation of low-income service work in the `on-demand' economy. Information, Communication & Society, Vol. 20, 6 (2017), 898--914.

[82]

Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work (CSCW '15). ACM, New York, NY, USA, 405--416.

Digital Library

[83]

Mark E Whiting, Dilrukshi Gamage, Snehalkumar S Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, et almbox. 2016. Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. arXiv preprint arXiv:1611.01572 (2016).

Digital Library

[84]

Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 379--388.

Digital Library

[85]

Bill Williamson. 1998. Lifeworlds and Learning. Essays in the Theory, Philosophy and Practice of Lifelong Learning. (1998).

[86]

Yu-Chun Grace Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social Network, Web Forum, or Task Market?: Comparing Different Crowd Genres for Design Feedback Exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 773--784.

Digital Library

[87]

Dongwook Yoon, Nicholas Chen, Bernie Randles, Amy Cheatle, Corinna E. Löckenhoff, Steven J. Jackson, Abigail Sellen, and Franccois Guimbretière. 2016. RichReview

[88]

: Deployment of a Collaborative Multi-modal Annotation System for Instructor Feedback and Peer Discussion. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 195--205.

Digital Library

[89]

Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445--1455.

Digital Library

[90]

Haiyi Zhu, Amy Zhang, Jiping He, Robert E. Kraut, and Aniket Kittur. 2013. Effects of Peer Feedback on Contribution: A Field Experiment in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2253--2262.

Digital Library

Cited By

Toxtli CCurtis CSavage S(2024)A Culturally-Aware AI Tool for Crowdworkers: Leveraging Chronemics to Support Diverse Work StylesProceedings of the ACM on Human-Computer Interaction10.1145/36868998:CSCW2(1-34)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686899
Bhat AKulkarni NHusain SYadavalli AKaur JShukla AShelar MSeshadri V(2024)Speaking in Terms of Money: Financial Knowledge Acquisition via Speech Data GenerationACM Journal on Computing and Sustainable Societies10.1145/36637752:3(1-35)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3663775
Do KDe Los Santos MMuller MSavage S(2024)Designing Gig Worker Sousveillance ToolsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642614(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642614
Show More Cited By

Index Terms

Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools

Recommendations

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Efficient Crowd Exploration of Large Networks: The Case of Causal Attribution

Accurately and efficiently crowdsourcing complex, open-ended tasks can be difficult, as crowd participants tend to favor short, repetitive "microtasks". We study the crowdsourcing of large networks where the crowd provides the network topology via ...
A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

This paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 2, Issue CSCW

November 2018

4104 pages

EISSN:2573-0142

DOI:10.1145/3290265

Editors:
Karrie Karahalios
University of Illinois & Adobe
,
Andrés Monroy-Hernández
Snap Inc.
,
Airi Lampinen
Stockholm University
,
Geraldine Fitzpatrick
Vienna University of Technology

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2018

Published in PACMHCI Volume 2, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

FY2018 Research Challenge Grant award
J. Wayne and Kathy Richards Faculty Fellowship in Engineering
Research Grant from Leidos Laboratories

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
549
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)3

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Toxtli CCurtis CSavage S(2024)A Culturally-Aware AI Tool for Crowdworkers: Leveraging Chronemics to Support Diverse Work StylesProceedings of the ACM on Human-Computer Interaction10.1145/36868998:CSCW2(1-34)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686899
Bhat AKulkarni NHusain SYadavalli AKaur JShukla AShelar MSeshadri V(2024)Speaking in Terms of Money: Financial Knowledge Acquisition via Speech Data GenerationACM Journal on Computing and Sustainable Societies10.1145/36637752:3(1-35)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3663775
Do KDe Los Santos MMuller MSavage S(2024)Designing Gig Worker Sousveillance ToolsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642614(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642614
Flores-Saviaga CCurtis CSavage S(2023)Inclusive Portraits: Race-Aware Human-in-the-Loop TechnologyProceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization10.1145/3617694.3623235(1-11)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3617694.3623235
De Los Santos MChávez NNavarrete AMartínez Pinto CGonzález LTelles-Calderon TSavage S(2023)La Independiente: Designing Ubiquitous Systems for Latin American and Caribbean Women CrowdworkersAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610728(404-406)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610728
Nouri ZPrakash NGadiraju UWachsmuth H(2023)Supporting Requesters in Writing Clear Crowdsourcing Task Descriptions Through Computational Flaw AssessmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584039(737-749)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584039
Huang ERees Lewis DGaudani SEasterday MGerber E(2023)Intelligent Coaching Systems: Understanding One-to-many Coaching for Ill-defined Problem SolvingProceedings of the ACM on Human-Computer Interaction10.1145/35796147:CSCW1(1-24)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579614
Lei KYu MLewellen MKu VLee D(2023)Compass: Supporting Large Group Mentorship in a Chat-Based UIProceedings of the ACM on Human-Computer Interaction10.1145/35794707:CSCW1(1-25)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579470
Calacci DPentland A(2022)Bargaining with the Black-BoxProceedings of the ACM on Human-Computer Interaction10.1145/35706016:CSCW2(1-24)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3570601
Stewart R(2022)Authenticity for Rent? Airbnb Hosts and the Commodification of Urban DisplacementProceedings of the ACM on Human-Computer Interaction10.1145/35555946:CSCW2(1-28)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555594
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents