research-article

Open access

Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

Pages 1902 - 1913

https://doi.org/10.1145/2998181.2998234

Published: 25 February 2017 Publication History

Abstract

Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.

Supplementary Material

ZIP File (cscwp0290-file4.zip)

Download
484.37 KB

References

[1]

Anderson, M. Crowdsourcing higher education: A design proposal for distributed learning. MERLOT Journal of Online Learning and Teaching, 7(4):576--590, 2011.

[2]

Bernstein, M., et al. Soylent. In Proceedings of the 23nd annual ACM symposium on User interface software and technology-UIST'10, p. 313. 2010.

Digital Library

[3]

Billett, S. Learning in the workplace: Strategies for effective practice. ERIC, 2001.

[4]

Boud, D. Sustainable assessment: rethinking assessment for the learning society. Studies in continuing education, 22(2):151--167, 2000.

[5]

Boud, D. et al. Enhancing learning through self-assessment. Routledge, 2013.

[6]

Callison-Burch, C. Fast, cheap, and creative: evaluating translation quality using amazon's mechanical turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pp. 286--295. Association for Computational Linguistics, 2009.

Digital Library

[7]

Campbell, J.A., et al. Thousands of positive reviews: Distributed mentoring in online fan communities. arXiv preprint arXiv:1510.01425, 2015.

Digital Library

[8]

Cheng, J., Teevan, J., and Bernstein, M.S. Measuring crowdsourcing effort with error-time curves. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1365--1374. ACM, 2015.

Digital Library

[9]

Deng, X.N. and Joshi, K. Is crowdsourcing a source of worker empowerment or exploitation? understanding crowd workers' perceptions of crowdsourcing career. 2013.

[10]

Dontcheva, M., Morris, R.R., Brandt, J.R., and Gerber, E.M. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pp. 3379--3388. ACM, 2014.

Digital Library

[11]

Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp. 1013--1022. ACM, 2012.

Digital Library

[12]

Ducheneaut, N., Yee, N., Nickell, E., and Moore, R.J. The life and death of online gaming communities: a look at guilds in world of warcraft. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 839--848. ACM, 2007.

Digital Library

[13]

Falchikov, N. and Goldfinch, J. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research, 70(3):287--322, 2000.

[14]

Gaikwad, S., et al. Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 625--637. ACM, 2016.

Digital Library

[15]

Gaikwad, S.N., et al. Daemo: A self-governed crowdsourcing marketplace. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, pp. 101--102. ACM, 2015.

Digital Library

[16]

Gilbert, E. What if we ask a different question?: social inferences create product ratings faster. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2759--2762. ACM, 2014.

Digital Library

[17]

Gray, M.L., Suri, S., Ali, S.S., and Kulkarni, D. The crowd is a collaborative network. Proceedings of Computer-Supported Cooperative Work, 2016.

Digital Library

[18]

Gupta, N., Martin, D., Hanrahan, B.V., and O'Neill, J. Turk-life in india. In Proceedings of the 18th International Conference on Supporting Group Work, pp. 1--11. ACM, 2014.

Digital Library

[19]

Guthrie, C. On learning the research craft: Memoirs of a journeyman researcher. Journal of Research Practice, 3(1):1, 2007.

[20]

Haas, D., Ansel, J., Gu, L., and Marcus, A. Argonaut: macrotask crowdsourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12):1642--1653, 2015.

Digital Library

[21]

Hata, K., Krishna, R., Fei-Fei, L., and Bernstein, M.S. A glimpse far into the future: Understanding long-term crowd worker accuracy. arXiv preprint arXiv:1609.04855, 2016.

Digital Library

[22]

Horton, J. and Golden, J. Reputation inflation: Evidence from an online labor market. Work. Pap., NYU, 2015.

[23]

Ipeirotis, P.G. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2):16--21, 2010.

Digital Library

[24]

Irani, L.C. and Silberman, M. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 611--620. ACM, 2013.

Digital Library

[25]

Jøsang, A., Ismail, R., and Boyd, C. A survey of trust and reputation systems for online service provision. Decision support systems, 43(2):618--644, 2007.

Digital Library

[26]

Kang, J., Ko, I., and Ko, Y. The impact of social support of guild members and psychological factors on flow and game loyalty in mmorpg. In System Sciences, 2009. HICSS'09. 42nd Hawaii International Conference on, pp. 1--9. IEEE, 2009.

Digital Library

[27]

Karoly, L.A. and Panis, C.W. The 21st century at work: Forces shaping the future workforce and workplace in the USA, vol. 164. Rand Corporation, 2004.

[28]

Kaye, L.K. and Bryce, J. Putting the fun factor into gaming: The influence of social contexts on the experiences of playing videogames. International Journal of Internet Science, 7(1):24--38, 2012.

[29]

Kieser, A. Organizational, institutional, and societal evolution: Medieval craft guilds and the genesis of formal organizations. Administrative Science Quarterly, pp. 540--564, 1989.

[30]

Kittur, A., et al. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, pp. 1301--1318. ACM, 2013.

Digital Library

[31]

Kulkarni, A., et al. Mobileworks: Designing for quality in a managed crowdsourcing architecture. Internet Computing, IEEE, 16(5):28--35, 2012.

Digital Library

[32]

Kulkarni, C., et al. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI), 20(6):33, 2013.

Digital Library

[33]

Kulkarni, C.E., Bernstein, M.S., and Klemmer, S.R. Peerstudio: Rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale, pp. 75--84. ACM, 2015.

Digital Library

[34]

LaPlante, R. and Silberman, M.S. Building trust in crowd worker forums: Worker ownership, governance, and work outcomes. In Proceedings of WebSci16. ACM, 2016.

[35]

Larson, M.S. and Larson, M.S. The rise of professionalism: A sociological analysis, vol. 233. Univ of California Press, 1979.

[36]

Laubacher, R.J., Malone, T.W., et al. Flexible work arrangements and 21st century worker's guilds. Tech. rep., MIT Center for Coordination Science, 1997.

[37]

Lee, H. and Choi, B. Knowledge management enablers, processes, and organizational performance: An integrative view and empirical examination. Journal of management information systems, 20(1):179--228, 2003.

Digital Library

[38]

Leskovec, J., Huttenlocher, D., and Kleinberg, J. Signed networks in social media. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1361--1370. ACM, 2010.

Digital Library

[39]

Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD workshop on human computation, pp. 68--76. ACM, 2010.

Digital Library

[40]

Little, G., Chilton, L.B., Goldman, M., and Miller, R.C. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 57--66. ACM, 2010.

Digital Library

[41]

Luther, K., et al. Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 473--485. ACM, 2015.

Digital Library

[42]

Malone, T.W. and Laubacher, R.J. How will work change? elancers, empowerment, and guilds. THE PROMISE OF GLOBAL NETWORKS, p. 119, 1999.

[43]

Martin, D., Hanrahan, B.V., O'Neill, J., and Gupta, N. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 224--235. ACM, 2014.

Digital Library

[44]

McInnis, B., Cosley, D., Nam, C., and Leshed, G. Taking a hit: Designing around rejection, mistrust, risk, and workers' experiences in amazon mechanical turk. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2271--2282. ACM, 2016.

Digital Library

[45]

Mitra, T., Hutto, C.J., and Gilbert, E. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1345--1354. ACM, 2015.

Digital Library

[46]

Mocarelli, L. Guilds reappraised: Italy in the early modern period. International review of social history, 53(S16):159--178, 2008.

[47]

Ogilvie, S. Guilds, efficiency, and social capital: evidence from german proto-industry. Economic history review, pp. 286--333, 2004.

[48]

Ogilvie, S. The use and abuse of trust: the deployment of social capital by early modern guilds. Jahrbuch für Wirtschaftsgeschichte, 1:15--52, 2005.

[49]

Patel, N., et al. Power to the peers: authority of source effects for a voice-based agricultural information service in rural india. In Proceedings of the fifth International Conference on Information and Communication Technologies and Development, pp. 169--178. ACM, 2012.

Digital Library

[50]

Peng Dai, M.D. and Weld, S. Decision-theoretic control of crowd-sourced workflows. In In the 24th AAAI Conference on Artificial Intelligence (AAAI'10). Citeseer, 2010.

Digital Library

[51]

Pérez, L. Inventing in a world of guilds: silk fabrics in eighteenth-century lyon. Guilds, innovation, and the European economy, pp. 1400--1800, 2008.

[52]

Poor, N. What mmo communities don't do: A longitudinal study of guilds and character leveling, or not. In Ninth International AAAI Conference on Web and Social Media. 2015.

[53]

Rees Lewis, D., Harburg, E., Gerber, E., and Easterday, M. Building support tools to connect novice designers with professional coaches. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, pp. 43--52. ACM, 2015.

Digital Library

[54]

Renard, G. Guilds in the middle ages. 1918.

[55]

Rosser, G. Crafts, guilds and the negotiation of work in the medieval town. Past & Present, (154):3--31, 1997.

[56]

Rzeszotarski, J.M. and Kittur, A. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 13--22. ACM, 2011.

Digital Library

[57]

Salehi, N., et al. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1621--1630. ACM, 2015.

Digital Library

[58]

Shaw, A. and Hill, B.M. Laboratories of oligarchy? how the iron law extends to peer production. Journal of Communication, 64(2):215--238, 2014.

[59]

Sheng, V.S., Provost, F., and Ipeirotis, P.G. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 614--622. ACM, 2008.

Digital Library

[60]

Sparrowe, R.T., Liden, R.C., Wayne, S.J., and Kraimer, M.L. Social networks and the performance of individuals and groups. Academy of management journal, 44(2):316--325, 2001.

[61]

Surowiecki, J. The wisdom of crowds. Anchor, 2005.

Digital Library

[62]

Suzuki, R., et al. Atelier: Repurposing expert crowdsourcing tasks as micro-internships. arXiv preprint arXiv:1602.06634, 2016.

Digital Library

[63]

Topping, K. Peer assessment between students in colleges and universities. Review of educational Research, 68(3):249--276, 1998.

[64]

Ulibarri, N., et al. Research as design: Developing creative confidence in doctoral students through design thinking. International Journal of Doctoral Studies, 9:249--270, 2014.

[65]

Wenger, E. Communities of practice: A brief introduction. 2011.

[66]

Yin, M., Gray, M.L., Suri, S., and Vaughan, J.W. The communication network within the crowd. In Proceedings of the 25th International Conference on World Wide Web, pp. 1293--1303. International World Wide Web Conferences Steering Committee, 2016.

Digital Library

[67]

Yu, L., André, P., Kittur, A., and Kraut, R. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 967--978. ACM, 2014.

Digital Library

[68]

Zhu, H., Dow, S.P., Kraut, R.E., and Kittur, A. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 1445--1455. ACM, 2014.

Digital Library

Cited By

Boone ARothschild AKoo XPfohl GSheehan ADiSalvo BLe Dantec CDiSalvo C(2024)Reimagining Meaningful Data Work through Citizen ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36870498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687049
Yang TMiceli M(2024)"Guilds" as Worker Empowerment and Control in a Chinese Data Work PlatformProceedings of the ACM on Human-Computer Interaction10.1145/36869048:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686904
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Show More Cited By

Index Terms

Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
1. Human-centered computing
  1. Collaborative and social computing

Recommendations

Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from ...
Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms
Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a ...
Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

Real-time collaboration between a requester and crowd workers expands the scope of tasks that crowdsourcing can be used for by letting requesters and crowd workers interactively create various artifacts (e.g., a sketch prototype, writing, or program code)...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

February 2017

2556 pages

ISBN:9781450343350

DOI:10.1145/2998181

General Chairs:
Charlotte P. Lee
University of Washington
,
Steve Poltrock
Retired
,
Program Chairs:
Louise Barkhuus
Stockholm University and Cornell Tech
,
Marcos Borges
Universidade Federal do Rio de Janeiro
,
Wendy Kellogg
IBM T.J. Watson Research Center

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Hasso Plattner Design Thinking Research Program
Office of Naval Research
Toyota Motor Corporation
National Science Foundation

Conference

CSCW '17

Sponsor:

SIGCHI

CSCW '17: Computer Supported Cooperative Work and Social Computing

February 25 - March 1, 2017

Oregon, Portland, USA

Acceptance Rates

CSCW '17 Paper Acceptance Rate 183 of 530 submissions, 35%;

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Sponsor:
sigchi

Computer-Supported Cooperative Work and Social Computing

October 18 - 22, 2025

Bergen , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
1,358
Total Downloads

Downloads (Last 12 months)163
Downloads (Last 6 weeks)29

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Boone ARothschild AKoo XPfohl GSheehan ADiSalvo BLe Dantec CDiSalvo C(2024)Reimagining Meaningful Data Work through Citizen ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36870498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687049
Yang TMiceli M(2024)"Guilds" as Worker Empowerment and Control in a Chinese Data Work PlatformProceedings of the ACM on Human-Computer Interaction10.1145/36869048:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686904
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Kashima HOyama SArai HMori J(2024)Trustworthy human computation: a surveyArtificial Intelligence Review10.1007/s10462-024-10974-157:12Online publication date: 12-Oct-2024
https://doi.org/10.1007/s10462-024-10974-1
Xia H(2023)Tragedy of the Commons in Crowd Work-Based ResearchACM Journal on Responsible Computing10.1145/36264931:1(1-25)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3626493
Lykourentzou IVinella FAhmed FPapastathis CPapangelis KKhan VMasthoff J(2022)Self-organization in online collaborative work settingsCollective Intelligence10.1177/263391372210780051:1(263391372210780)Online publication date: 9-Sep-2022
https://doi.org/10.1177/26339137221078005
Huang KZhou JChen S(2022)Being a Solo Endeavor or Team Worker in Crowdsourcing Contests? It is a Long-term Decision You Need to MakeProceedings of the ACM on Human-Computer Interaction10.1145/35555956:CSCW2(1-32)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555595
Rechkemmer AYin M(2022)Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative ViewProceedings of the ACM on Human-Computer Interaction10.1145/35551376:CSCW2(1-30)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555137
Alvarez de la Vega JCecchinato MRooksby J(2022)Design Opportunities for Freelancing Platforms: Online Freelancers’ Views on a Worker-Centred Design FictionProceedings of the 1st Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3533406.3533410(1-19)Online publication date: 8-Jun-2022
https://dl.acm.org/doi/10.1145/3533406.3533410
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents