research-article

Paying Crowd Workers for Collaborative Work

Authors:

Edith LawAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 3, Issue CSCW

Article No.: 125, Pages 1 - 24

https://doi.org/10.1145/3359227

Published: 07 November 2019 Publication History

Abstract

Collaborative crowdsourcing tasks allow crowd workers to solve problems that they could not handle alone, but worker motivation in these tasks is not well understood. In this paper, we study how to motivate groups of workers by paying them equitably. To this end, we characterize existing collaborative tasks based on the types of information available to crowd workers. Then, we apply concepts from equity theory to show how fair payments relate to worker motivation, and we propose two theoretically grounded classes of fair payments. Finally, we run two experiments using an audio transcription task on Amazon Mechanical Turk to understand how workers perceive these payments. Our results show that workers recognize fair and unfair payment divisions, but are biased toward payments that reward them more. Additionally, our data suggests that fair payments could lead to a small increase in worker effort. These results inform the design of future collaborative crowdsourcing tasks.

References

[1]

John Stacy Adams. 1965. Inequity In Social Exchange. In Advances in Experimental Social Psychology. Vol. 2. 267--299. https://doi.org/10.1016/S0065--2601(08)60108--2

[2]

Michael S Bernstein, Joel Brandt, Robert C Miller, and David R Karger. 2011. Crowds in two seconds: enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. ACM Press, New York, New York, USA, 33. https://doi.org/10.1145/2047196.2047201

Digital Library

[3]

Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A Word Processor with a Crowd Inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology - UIST '10. ACM Press, New York, New York, USA, 313. https://doi.org/10.1145/1866029.1866078

Digital Library

[4]

Georgios Chalkiadakis, Edith Elkind, and Michael Wooldridge. 2011. Computational Aspects of Cooperative Game Theory. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, 6 (oct 2011), 1--168. https://doi.org/10. 2200/S00355ED1V01Y201107AIM016

[5]

Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17. ACM Press, New York, New York, USA, 2334--2346. https://doi.org/10.1145/3025453.3026044

Digital Library

[6]

Quanze Chen, Jonathan Bragg, Lydia B. Chilton, and Daniel S. Weld. 2019. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. In Proceedings of the 2019 ACM annual conference on Human Factors in Computing Systems - CHI '19.

[7]

D. Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A. Hearst. 2015. Structuring Interactions for Large-Scale Synchronous Peer Learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW '15. ACM Press, New York, New York, USA, 1139--1152. https: //doi.org/10.1145/2675133.2675251

[8]

Jason A. Colquitt. 2001. On the dimensionality of organizational justice: A construct validation of a measure. Journal of Applied Psychology 86, 3 (2001), 386--400. https://doi.org/10.1037//0021--9010.86.3.386

[9]

J H Davis, P R Laughlin, and S S Komorita. 1976. The Social Psychology of Small Groups: Cooperative and Mixed-Motive Interaction. Annual Review of Psychology 27, 1 (1976), 501--541. https://doi.org/10.1146/annurev.ps.27.020176.002441 arXiv:https://doi.org/10.1146/annurev.ps.27.020176.002441

[10]

Ryan Drapeau, Lydia B Chilton, and Daniel S Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016) (2016), 32--41.

[11]

Christian Fieseler, Eliane Bucher, and Christian Pieter Hoffmann. 2017. Unfairness by Design? The Perceived Fairness of Digital Labor on Crowdworking Platforms. Journal of Business Ethics (jun 2017). https://doi.org/10.1007/ s10551-017--3607--2

[12]

Ya'akov (Kobi) Gal, Moshe Mash, Ariel D. Procaccia, and Yair Zick. 2016. Which Is the Fairest (Rent Division) of Them All?. In Proceedings of the 2016 ACM Conference on Economics and Computation - EC '16. ACM Press, New York, New York, USA, 67--84. https://doi.org/10.1145/2940716.2940724

Digital Library

[13]

Mary L Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The Crowd is a Collaborative Network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW '16. ACM Press, New York, New York, USA, 134--147. https://doi.org/10.1145/2818048.2819942

Digital Library

[14]

Nathan Hahn, Joseph Chang, Ji Eun Kim, and Aniket Kittur. 2016. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 2258--2270. https://doi.org/10.1145/2858036.2858364

Digital Library

[15]

Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--14. https://doi.org/10.1145/ 3173574.3174023 arXiv:1712.05796

[16]

Joseph W. Harder. 1992. Play for Pay: Effects of Inequity in a Pay-for-Performance Context. Administrative Science Quarterly 37, 2 (1992), 321--335. http://www.jstor.org/stable/2393227

[17]

Fredrik Heyman. 2005. Pay inequality and firm performance: evidence from matched employer employee data. Applied Economics 37, 11 (2005), 1313--1327. https://doi.org/10.1080/00036840500142101 arXiv:https://doi.org/10.1080/00036840500142101

[18]

Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015. Incentivizing High Quality Crowdwork. In Proceedings of the 24th International Conference on World Wide Web - WWW '15. ACM Press, New York, New York, USA, 419--429. https://doi.org/10.1145/2736277.2741102

Digital Library

[19]

Shih-Wen Huang and Wai-Tat Fu. 2013. Don't hide in the crowd!: increasing social transparency between peer workers improves crowdsourcing outcomes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM Press, New York, New York, USA, 621. https://doi.org/10.1145/2470654.2470743

Digital Library

[20]

Ting-Hao Kenneth Huang, Joseph Chee Chang, Saiganesh Swaminathan, and Jeffrey P. Bigham. 2017. Evorus: A Crowd-powered Conversational Assistant That Automates Itself Over Time. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17. ACM Press, New York, New York, USA, 155--157. https://doi.org/10.1145/3131785.3131823

[21]

Ting-Hao (Kenneth) Huang, Walter S. Lasecki, Amos Azaria, and Jeffrey P. Bigham. 2016. "Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing. 79--88.

[22]

Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM Press, New York, New York, USA, 611. https://doi.org/10.1145/2470654.2470742

Digital Library

[23]

Reinoud Joosten. 1996. Dynamics, equilibria, and values. Ph.D. Dissertation. Maastricht University.

[24]

Alexandre Kaspar, Genevieve Patterson, Changil Kim, Yagiz Aksoy, Wojciech Matusik, and Mohamed Elgharib. 2018. Crowd-Guided Ensembles: How Can We Choreograph Crowd Workers for Video Segmentation?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--12. https://doi.org/10.1145/3173574.3173685

Digital Library

[25]

Nicolas Kaufmann, Thimo Schulze, and Daniel Viet. 2011. More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk. In Proceedings of the Seventeenth Americas Conference on Information Systems. ACM Press, New York, New York, USA.

[26]

Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P. Bigham, Jaime Teevan, Ece Kamar, and Walter S. Lasecki. 2017. CrowdMask: Using Crowds to Preserve Privacy in Crowd-Powered Systems via Progressive Filtering. In Fifth AAAI Conference on Human Computation and Crowdsourcing. 89--97.

[27]

Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, and Michael S. Bernstein. 2017. Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17. ACM Press, New York, New York, USA, 233--245. https://doi.org/ 10.1145/2998181.2998196

[28]

Joy O Kim and Andres Monroy-Hernandez. 2016. Storia: Summarizing Social Media Content based on Narrative Theory using Crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW '16. ACM Press, New York, New York, USA, 1016--1025. https://doi.org/10.1145/2818048.2820072

Digital Library

[29]

Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. ACM Press, New York, New York, USA, 43. https://doi.org/10.1145/2047196.2047202

Digital Library

[30]

Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work - CSCW '12. ACM Press, New York, New York, USA, 1003. https://doi.org/10.1145/2145204.2145354

Digital Library

[31]

Anand Pramod Kulkarni, Matthew Can, and Björn Hartmann. 2011. Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk. In CHI'11 Extended Abstracts on Human Factors in Computing Systems.

[32]

Thierry Lallemand, Robert Plasman, and François Rycx. 2009. Wage structure and firm productivity in Belgium. In The structure of wages: An international comparison. University of Chicago Press, 179--215.

[33]

Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12. ACM Press, New York, New York, USA, 23. https://doi.org/10.1145/ 2380116.2380122

Digital Library

[34]

Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces that Come to Life as You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, New York, New York, USA, 1925--1934. https: //doi.org/10.1145/2702123.2702565

[35]

Walter S Lasecki, Kyle I Murray, Samuel White, Robert C Miller, and Jeffrey P Bigham. 2011. Real-time crowd control of existing interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. ACM Press, New York, New York, USA, 23. https://doi.org/10.1145/2047196.2047200

Digital Library

[36]

Jesse D. Lecy and Kate E. Beatty. 2012. Representative Literature Reviews Using Constrained Snowball Sampling and Citation Network Analysis. SSRN Electronic Journal (2012). https://doi.org/10.2139/ssrn.1992601

[37]

Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP '10. ACM Press, New York, New York, USA, 68. https://doi.org/10.1145/1837885.1837907

[38]

Weichen Liu, Sijia Xiao, Jacob T. Browne, Ming Yang, and Steven P. Dow. 2018. ConsensUs: Supporting Multi-Criteria Group Decisions by Visualizing Points of Disagreement. ACM Transactions on Social Computing 1, 1 (jan 2018), 1--26. https://doi.org/10.1145/3159649

Digital Library

[39]

Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and Steven P. Dow. 2016. Personality Matters: Balancing for Personality Types Leads to Better Outcomes for Crowd Teams. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 260--273. https: //doi.org/10.1145/2818048.2819979

[40]

Ioanna Lykourentzou, Robert E Kraut, and Steven P Dow. 2017. Team Dating Leads to Better Online Ad Hoc Collaborations. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17. ACM Press, New York, New York, USA, 2330--2343. https://doi.org/10.1145/2998181.2998322

Digital Library

[41]

Thomas W. Malone and Kevin Crowston. 1990. What is Coordination Theory and How Can It Help Design Cooperative Work Systems?. In Proceedings of the 1990 ACM Conference on Computer-supported Cooperative Work (CSCW '90). ACM, New York, NY, USA, 357--370. https://doi.org/10.1145/99332.99367

[42]

Thomas W. Malone, Thomas W. Malone, and Kevin Crowston. 1994. The Interdisciplinary Study of Coordination. ACM Comput. Surv. 26, 1 (March 1994), 87--119. https://doi.org/10.1145/174666.174668

Digital Library

[43]

Andrew Mao, Winter Mason, Siddharth Suri, and Duncan J. Watts. 2016. An Experimental Study of Team Size and Performance on a Complex Task. PLOS ONE 11, 4 (apr 2016), e0153048. https://doi.org/10.1371/journal.pone.0153048

[44]

David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing - CSCW '14. ACM Press, New York, New York, USA, 224--235. https://doi.org/10.1145/2531602.2531663

Digital Library

[45]

David Martin, Jacki O'Neill, Neha Gupta, and Benjamin V. Hanrahan. 2016. Turking in a Global Labour Market. Computer Supported Cooperative Work (CSCW) 25, 1 (01 Feb 2016), 39--77. https://doi.org/10.1007/s10606-015--9241--6

[46]

Winter Mason and Duncan J Watts. 2009. Financial incentives and the "performance of crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP '09. ACM Press, New York, New York, USA, 77. https://doi.org/10.1145/1600150.1600175

Digital Library

[47]

J.E. McGrath. 1984. Groups: Interaction and Performance. Prentice-Hall. https://books.google.ca/books?id= 4pzZAAAAMAAJ

[48]

David M. Messick and Keith P. Sentis. 1979. Fairness and preference. Journal of Experimental Social Psychology 15, 4 (1979), 418--434. https://doi.org/10.1016/0022--1031(79)90047--7

[49]

Nolan Miller, Paul Resnick, and Richard Zeckhauser. 2005. Eliciting Informative Feedback: The Peer-Prediction Method. Management Science 51, 9 (2005), 1359--1373. https://doi.org/10.1287/mnsc.1050.0379

Digital Library

[50]

Meredith Ringel Morris, Jeffrey P. Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting Microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17. ACM Press, New York, New York, USA, 1867--1876. https://doi.org/10.1145/3025453.3025687

[51]

Richard T. Mowday. 1979. Equity Theory Predictions of Behavior in Organizations. In Motivation and Work Behavior (2 ed.), Richard M. Steers and Lyman W. Porter (Eds.). Mcgraw-Hill, New York, 111--131.

[52]

Eugene W. Myers. 1986. An O(ND) difference algorithm and its variations. Algorithmica 1, 1--4 (1986), 251--266. https://doi.org/10.1007/BF01840446

Digital Library

[53]

Gary M. Olson and Judith S. Olson. 2000. Distance Matters. Hum.-Comput. Interact. 15, 2 (Sept. 2000), 139--178. https://doi.org/10.1207/S15327051HCI1523_4

Digital Library

[54]

Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. ICWSM International Conference on Web and Social Media (2011), 321--328. https://doi.org/10.13140/RG.2.2.19170.94401

[55]

Michael Ross and Fiore Sicoly. 1979. Egocentric biases in availability and attribution., 322--336 pages. https: //doi.org/10.1037/0022--3514.37.3.322

[56]

Niloufar Salehi, Andrew McCabe, Melissa Valentine, and Michael Bernstein. 2017. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17. ACM Press, New York, New York, USA, 1700--1713. https://doi.org/10.1145/2998181.2998300

Digital Library

[57]

Shikhar Sarin and Vijay Mahajan. 2001. The Effect of Reward Structures on the Performance of Cross-Functional Product Development Teams. Journal of Marketing 65, 2 (2001), 35--53. https://doi.org/10.1509/jmkg.65.2.35.18252 arXiv:https://doi.org/10.1509/jmkg.65.2.35.18252

[58]

Mike Schaekermann, Joslin Goh, Kate Larson, and Edith Law. 2018. Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (nov 2018), 1--19. https://doi.org/10.1145/3274423

Digital Library

[59]

Pao Siangliulue, Kenneth C. Arnold, Krzysztof Z. Gajos, and Steven P. Dow. 2015. Toward Collaborative Ideation at Scale: Leveraging Ideas from Others to Generate More Creative and Diverse Ideas. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW '15. ACM Press, New York, New York, USA, 937--945. https://doi.org/10.1145/2675133.2675239

[60]

Maximilian Speicher and Michael Nebeling. 2018. GestureWiz: A Human-Powered Gesture Design Environment for User Interface Prototypes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--11. https://doi.org/10.1145/3173574.3173681

Digital Library

[61]

Leigh Thompson and George Loewenstein. 1992. Egocentric perceptions of fairness and interpersonal conflict. Organizational Behavior & Human Decision Processes 51 (1992), 176--197.

[62]

Ming Yin, Yiling Chen, and Yu-An Sun. 2013. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence.

Digital Library

[63]

Ming Yin, Mary L. Gray, Siddharth Suri, and Jennifer Wortman Vaughan. 2016. The Communication Network Within the Crowd. In Proceedings of the 25th International Conference on World Wide Web - WWW '16. ACM Press, New York, New York, USA, 1293--1303. https://doi.org/10.1145/2872427.2883036

Digital Library

[64]

Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12. ACM Press, New York, New York, USA, 217. https://doi.org/10.1145/2207676.2207708

Digital Library

[65]

Sharon Zhou, Melissa Valentine, and Michael S Bernstein. 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--13. https://doi.org/10.1145/3173574.3173682

Digital Library

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Shelby RSrinivasan RBurgdorf KLena JRostamzadeh N(2024)Creative ML Assemblages: The Interactive Politics of People, Processes, and ProductsProceedings of the ACM on Human-Computer Interaction10.1145/36373158:CSCW1(1-30)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637315
Kashima HOyama SArai HMori J(2024)Trustworthy human computation: a surveyArtificial Intelligence Review10.1007/s10462-024-10974-157:12Online publication date: 12-Oct-2024
https://doi.org/10.1007/s10462-024-10974-1
Show More Cited By

Index Terms

Paying Crowd Workers for Collaborative Work
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
    2. Empirical studies in collaborative and social computing

Recommendations

Quantifying the Invisible Labor in Crowd Work
CSCW2

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Building an Honest and Capable Crowd Workforce
SIGMIS-CPR'18: Proceedings of the 2018 ACM SIGMIS Conference on Computers and People Research

One of the major challenges of crowdsourcing is to build a labor force that is not only capable but also honest. Prior research on crowdsourcing-mechanism design mostly focuses on motivating individual worker's effort, while neglecting the importance of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 3, Issue CSCW

November 2019

5026 pages

EISSN:2573-0142

DOI:10.1145/3371885

Editors:
Airi Lampinen
Stockholm University, Sweden
,
Darren Gergle
Northwestern University, USA
,
David A. Shamma
FXPAL, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2019

Published in PACMHCI Volume 3, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSERC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
385
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)8

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Shelby RSrinivasan RBurgdorf KLena JRostamzadeh N(2024)Creative ML Assemblages: The Interactive Politics of People, Processes, and ProductsProceedings of the ACM on Human-Computer Interaction10.1145/36373158:CSCW1(1-30)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637315
Kashima HOyama SArai HMori J(2024)Trustworthy human computation: a surveyArtificial Intelligence Review10.1007/s10462-024-10974-157:12Online publication date: 12-Oct-2024
https://doi.org/10.1007/s10462-024-10974-1
Krendl AHugenberg KKennedy D(2023)Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasksBehavior Research Methods10.3758/s13428-023-02152-y56:3(2353-2375)Online publication date: 15-Jun-2023
https://doi.org/10.3758/s13428-023-02152-y
Maddalena EIbáñez LReeves NSimperl E(2023)Qrowdsmith: Enhancing Paid Microtask Crowdsourcing with Gamification and Furtherance IncentivesACM Transactions on Intelligent Systems and Technology10.1145/360494014:5(1-26)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3604940
Behl AJayawardena NPereira VJabeen FJain KGupta M(2023)Engaging and motivating crowd-workers in gamified crowdsourcing mobile apps in the context of logistics and sustainable supply chain managementAnnals of Operations Research10.1007/s10479-023-05557-2Online publication date: 22-Aug-2023
https://doi.org/10.1007/s10479-023-05557-2
Montanelli SRuskov M(2023)A Systematic Literature Review of Online Collaborative Story WritingHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42286-7_5(73-93)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-42286-7_5
Soratana TLiu YYang X(2022)Effects of Payment Rate and Country’s Income Level on Attitude Toward aCrowdsourcing TaskProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266153266:1(2220-2224)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661532
Soratana TLiu YJessie Yang X(2022)Effect of Payment Methods in Crowdsourcing PlatformsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/107118132266113566:1(1627-1631)Online publication date: 27-Oct-2022
https://doi.org/10.1177/1071181322661135
Wang YPapangelis KLykourentzou IKhan VSaker MYue YGrudin J(2022)The dawn of crowdfarmsCommunications of the ACM10.1145/349069865:8(64-70)Online publication date: 21-Jul-2022
https://dl.acm.org/doi/10.1145/3490698
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents