research-article

Work experiences on MTurk

Authors:

Alice M. Brawley,

Cynthia L.S. PuryAuthors Info & Claims

Computers in Human Behavior, Volume 54, Issue C

Pages 531 - 546

https://doi.org/10.1016/j.chb.2015.08.031

Published: 01 January 2016 Publication History

Abstract

Amazon's Mechanical Turk (MTurk) is an online marketplace for work, where Requesters post Human Intelligence Tasks (HITs) for Workers to complete for varying compensation. Past research has focused on the quality and generalizability of social and behavioral science research conducted using MTurk as a source of research participants. However, MTurk and other crowdsourcing platforms also exemplify trends toward extremely short-term contract work. We apply principles of industrial-organizational (I-O) psychology to investigate MTurk Worker job satisfaction, information sharing, and turnover. We also report the top best and worst Requester behaviors (e.g., building a relationship, unfair pay) that affect Worker satisfaction. Worker satisfaction was consistently negatively related to turnover as expected, indicating that this traditional variable operates similarly in the MTurk work context. However, few of the traditional predictors of job satisfaction were significant, signifying that new operational definitions or entirely new variables may be needed in order to adequately understand the experiences of crowdsourced workers. Coworker friendships consistently predicted information sharing among Workers. The findings of this study are useful for understanding the experiences of crowdsourced workers from the perspective of I-O psychology, as well as for researchers using MTurk as a recruitment tool. We studied the work experiences of MTurk Workers.Job satisfaction consistently predicted turnover.However, few traditional predictors of job satisfaction were significant.New definitions or constructs may be needed to study this work experience.We also report the top best and worst practices for MTurk Requesters.

References

[1]

H. Aguinis, S.O. Lawal, eLancing: a review and research agenda for bridging the science-practice gap, Human Resource Management Review, 23 (2013) 6-17.

[2]

J. Antin, A. Shaw, Social desirability bias and self-reports of motivation: a study of Amazon Mechanical Turk in the US and India, 2012.

[3]

P. Barger, T.S. Behrend, D.J. Sharek, E.F. Sinar, IO and the crowd: frequently asked questions about using Mechanical Turk for research, The Industrial-Organizational Psychologist, 49 (2011) 11-17.

[4]

R.F. Baumeister, E. Bratslavsky, C. Finkenauer, K.D. Vohs, Bad is stronger than good, Review of General Psychology, 5 (2001) 323-370.

[5]

B.B. Bederson, A.J. Quinn, Web workers unite! addressing challenges of online laborers, 2011.

[6]

T.S. Behrend, D.J. Sharek, A.W. Meade, E.N. Wiebe, The viability of crowdsourcing for survey research, Behavior Research Methods, 43 (2011) 800-813.

[7]

A.M. Brawley, C.L.S. Pury, De-balkanization: a molar organizational climate framework based on situational affordances, in: Poster presented at the 29th annual meeting of the Society of Industrial and Organizational Psychology, Honolulu, HI, 2014.

[8]

S.P. Brown, R.A. Peterson, Antecedents and consequences of salesperson job satisfaction: meta-analysis and assessment of causal effects, Journal of Marketing Research, 30 (1993) 63-77.

[9]

M. Buhrmester, T. Kwang, S.D. Gosling, Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data?, Perspectives on Psychological Science, 6 (2011) 3-5.

[10]

S. Cartwright, New forms of work organization: Issues and challenges, Leadership & Organization Development Journal, 24 (2003) 121-122.

[11]

W.R. Cascio, H. Aguinis, Applied psychology in human resource management, Pearson, Upper Saddle River, NJ, 2011.

[12]

K. Casler, L. Bickel, E. Hackett, Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing, Computers in Human Behavior, 29 (2013) 2156-2160.

Digital Library

[13]

J. Chandler, P. Mueller, G. Paolacci, Nonnaïveté among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers, Behavior Research Methods, 46 (2014) 112-130.

[14]

J. Chandler, G. Paolacci, P. Mueller, Risks and rewards of crowdsourcing marketplaces, in: Handbook of human computation, Springer, New York, 2013, pp. 377-392.

[15]

W.H. Cooper, M.J. Withey, The strong situation hypothesis, Personality and Social Psychology Review, 13 (2009) 62-72.

[16]

R. Cropanzano, K. James, M.A. Konovsky, Dispositional affectivity as a predictor of work attitudes and job performance, Journal of Organizational Behavior, 14 (1993) 595-606.

[17]

M.B. Donnellan, F.L. Oswald, B.M. Baird, R.E. Lucas, The mini-IPIP scales: tiny-yet-effective measures of the big five factors of personality, Psychological Assessment, 18 (2006) 192-203.

[18]

J.K. Goodman, C.E. Cryder, A. Cheema, Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples, Journal of Behavioral Decision Making, 26 (2012) 213-224.

[19]

R.W. Griffeth, P.W. Hom, S. Gaertner, A¿meta-analysis of antecedents and correlates of employee turnover: update, moderator tests, and research implications for the next millenium, Journal of Management, 26 (2000) 463-488.

[20]

N. Gupta, A. Crabtree, T. Rodden, D. Martin, J. O'Neill, Understanding Indian crowdworkers, 2014.

[21]

J.R. Hackman, G.R. Oldham, The job diagnostic survey: an instrument for the diagnosis of jobs and the evaluation of job redesign projects: DTIC document, 1974.

[22]

D.J. Henderson, S.J. Wayne, L.M. Shore, W.H. Bommer, L.E. Tetrick, Leader-member exchange, differentiation, and psychological contract fulfillment: a multilevel examination, Journal of Applied Psychology, 93 (2008) 1208-1219.

[23]

W.A. Hochwarter, C. Kacmar, P.L. Perrewe, D. Johnson, Perceived organizational support as a mediator of the relationship between politics perceptions and work outcomes, Journal of Vocational Behavior, 63 (2003) 438-456.

[24]

C.J. Holden, T. Dennie, A.D. Hicks, Assessing the reliability of the M5-120 on Amazon's Mechanical Turk, Computers in Human Behavior, 29 (2013) 1749-1754.

[25]

J.J. Horton, The condition of the turking class: are online employers fair and honest?, Economics Letters, 111 (2011) 10-12.

[26]

C.J. Ho, A. Slivkins, S. Suri, J.W. Vaughan, Incentivizing high quality crowdwork, 2015.

[27]

P.G. Ipeirotis, Analyzing the Amazon Mechanical Turk marketplace, XRDS: Crossroads, The ACM Magazine for Students, 17 (2010) 16-21.

Digital Library

[28]

P.G. Ipeirotis, Demographics of mechanical turk, New York University, 2010. http://hdl.handle.net/2451/29585

[29]

L.C. Irani, The cultural work of microwork, New Media & Society, 17 (2015) 720-739.

[30]

L.C. Irani, Difference and dependence among digitial workers: the case of Amazon Mechanical Turk, South Atlantic Quarterly, 114 (2015) 225-234.

[31]

L.C. Irani, M.S. Silberman, Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk, 2013.

[32]

G.H. Ironson, P.C. Smith, M.T. Brannick, W.M. Gibson, K.B. Paul, Construction of a job in general scale: a comparison of global, composite and specific measures, Journal of Applied Psychology, 74 (1989) 193-200.

[33]

T.A. Judge, A. Erez, J.E. Bono, C.J. Thoresen, The core self-evaluations scale: development of a measure, Personnel Psychology, 56 (2003) 303-331.

[34]

T.A. Judge, D. Heller, R. Klinger, The dispositional sources of job satisfaction: a comparative test, Applied Psychology: An International Review, 57 (2008) 361-372.

[35]

T.A. Judge, J.D. Kammeyer-Mueller, Job attitudes, Annual Review of Psychology, 63 (2012) 341-367.

[36]

A. Kittur, J.V. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, The future of crowd work, 2013.

[37]

R.N. Landers, T.S. Behrend, An inconvenient truth: arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples, Industrial and Organizational Psychology: Perspectives on Science and Practice, 8 (2015) 142-164.

[38]

F.J. Landy, J.M. Conte, Work in the 21st century: an introduction to industrial and organizational psychology, John Wiley & Sons, Hoboken, NJ, 2009.

[39]

V. Lehdonvirta, P. Mezier, Identity and self-organization in unstructured work, The University of Hertfordshire, Hertfordshire AL10 9AB, UK, 2013.

[40]

C.J. Lemelle, S.A. Scielzo, How you feel about yourself can affect how¿you feel about your job: a meta-analysis examining the relationship of core self-evaluations and job satisfaction, Journal of Business Diversity, 12 (2012) 116-133.

[41]

P.E. Levy, Industrial/organizational psychology: Understanding the workplace, Houghton Mifflin Company, Boston, 2006.

[42]

E.F. Loftus, D.G. Miller, H.J. Burns, Semantic integration of verbal information into a visual memory, Journal of Experimental Psychology: Human Learning and Memory, 4 (1978) 19-31.

[43]

B.T. Loher, R.A. Noe, N.L. Moeller, M.P. Fitzgerald, A¿meta-analysis of the relation of job characteristics to job satisfaction, Journal of Applied Psychology, 70 (1985) 280-289.

[44]

C.P. Maertz, M.L. Campion, Profiles in quitting: Integrating process and content turnover theory, Academy of Management Journal, 47 (2004) 566-582.

[45]

C.C. Marshall, F.M. Shipman, Experiences surveying the crowd: Reflections on methods, participation, and reliability, 2013.

[46]

D. Martin, B.V. Hanrahan, J. O'Neill, N. Gupta, Being a turker, 2014.

[47]

W. Mason, S. Suri, Conducting behavioral research on Amazon's Mechanical Turk, Behavior Research Methods, 44 (2012) 1-23.

[48]

R.D. Meyer, R.S. Dalal, Situational strength as a means of conceptualizing context, Industrial and Organizational Psychology, 2 (2009) 99-102.

[49]

D.A. Newman, Longitudinal modeling with randomly and systematically missing data: a simulation of ad hoc, maximum likelihood, and multiple imputation techniques, Organizational Research Methods, 6 (2003) 328-362.

[50]

I.K. Nielsen, S.M. Jex, G.A. Adams, Development and validation of scores on a two-dimensional workplace friendship scale, Educational and Psychological Measurement, 60 (2000) 628-643.

[51]

J.C. Nunnally, Psychometric theory, McGraw-Hill Publishing, New York, 1978.

[52]

T.A. O'Neill, L.A. Hambley, N.S. Greidanus, R. MacDonnell, T.J.B. Kline, Predicting teleworker success: an exploration of personality, motivational, situational, and job characteristics, New Technology, Work, and Employment, 24 (2009) 144-162.

[53]

G. Paolacci, J. Chandler, Inside the turk: understanding Mechanical Turk as a participant pool, Current Directions in Psychological Science, 23 (2014) 184-188.

[54]

P. Paritosh, P. Ipeirotis, M. Cooper, S. Suri, The computer is the new sewing machine: Benefits and perils of crowdsourcing, 2011.

[55]

J. Ross, L.C. Irani, M.S. Silberman, A. Zaldivar, B. Tomlinson, Who are the crowdworkers?: Shifting demographics in Mechanical Turk, 2010.

[56]

A.M. Ryan, J.L. Wessel, Implications of a changing workforce and workplace for justice perceptions and expectations, Human Resource Management Review, 25 (2015) 162-175.

[57]

N. Salehi, L.C. Irani, M.S. Bernstein, A. Alkhatib, E. Ogbe, K. Milland, We are dynamo: Overcoming stalling and friction in collective action for crowd workers, 2015.

[58]

S.L. Sauter, W.S. Brightwell, M.J. Colligan, J.J. Hurrell, T.M. Katz, D.E. LeGrande, The changing organization of work and the safety and health of working people, National Institute for Occupational Safety and Health, Cincinnati, 2002.

[59]

J.L. Schafer, J.W. Graham, Missing data: our view of the state of the art, Psychological Methods, 7 (2002) 147-177.

[60]

G. Schmidt, Fifty days an MTurk Worker: the social and motivational context for Amazon Mechanical Turk Workers, Industrial and Organizational Psychology: Perspectives on Science and Practice, 8 (2015) 165-171.

[61]

A. Schriner, D. Oerther, No really, (crowd) work is the silver bullet, Procedia Engineering, 78 (2014) 224-228.

[62]

T. Schulze, S. Krug, M. Schader, Workers' task choice in crowdsourcing and human computation markets, 2012.

[63]

J.D. Shaw, M.K. Duffy, G.D. Jenkins, N. Gupta, Positive and negative affect, signal sensitivity, and pay satisfaction, Journal of Management, 25 (1999) 189-206.

[64]

M. Silberman, L. Irani, J. Ross, Ethics and tactics of professional crowdwork, XRDS: Crossroads, The ACM Magazine for Students, 17 (2010) 39-43.

Digital Library

[65]

M. Snyder, W. Ickes, Personality and social behaviour, in: Handbook of social psychology: Vol. 2, special fields and applications, Random House, New York, 1985, pp. 883-948.

[66]

M. Staffelbach, P. Sempolinski, D. Hachen, A. Kareem, T. Kijewski-Correa, D. Thain, Lessons learned from an experiment in crowdsourcing complex citizen engineering tasks with Amazon Mechanical Turk, 2014.

[67]

B.M. Staw, J. Ross, Stability in the midst of change: a dispositional approach to job attitudes, Journal of Applied Psychology, 70 (1985) 469-480.

[68]

A. Summerville, C.R. Chartier, Pseudo-dyadic "interaction" on Amazon's Mechanical Turk, Behavior Research Methods, 45 (2012) 116-124.

[69]

K.J. Templer, Five-Factor Model of personality and job satisfaction: the importance of agreeableness in a tight and collectivistic Asian society, Applied Psychology: An International Review, 61 (2012) 114-129.

[70]

R. Teodoro, P. Ozturk, M. Naaman, W. Mason, J. Lindqvist, The motivations and experiences of the on-demand mobile workforce, 2014.

[71]

L.E. Tetrick, K.J. Slack, N. Da Silva, R.R. Sinclair, A¿comparison of the stress-strain process for business owners and nonowners: differences in job demands, emotional exhaustion, satisfaction, and social support, Journal of Occupational Health Psychology, 5 (2000) 464-476.

[72]

R.P. Tett, J.P. Meyer, Job satisfaction, organizational commitment, turnover intention, and turnover: path analyses based on meta-analytic findings, Personnel Psychology, 46 (1993) 259-293.

[73]

E.R. Thompson, F.T.T. Phua, A¿brief index of affective job satisfaction, Group & Organization Management, 37 (2012) 275-307.

[74]

S. Tonidandel, J.M. LeBreton, Relative importance analysis: a useful supplement to regression analysis, Journal of Business and Psychology, 26 (2011) 1-9.

[75]

D. Vakharia, M. Lease, Beyond Mechanical Turk: an analysis of paid crowdwork platforms, 2015.

[76]

V.H. Vroom, Work and motivation, Wiley, New York, 1964.

[77]

D. Watson, L.A. Clark, A. Tellegen, Development and validation of brief measures of positive and negative affect: the PANAS scales, Journal of Personality and Social Psychology, 54 (1988) 1063-1070.

[78]

J. Webster, L.K. Trevino, L. Ryan, The dimensionality and correlates of flow in human-computer interactions, Computers in Human Behavior, 9 (1993) 411-426.

[79]

M.L. Williams, M.A. McDaniel, N.T. Nguyen, A¿meta-analysis of the antecedents and consequences of pay level satisfaction, Journal of Applied Psychology, 91 (2006) 392-413.

[80]

J. Zittrain, Ubiquitous human computing. Philosophical transactions of the royal society of London a: Mathematical, Physical and Engineering Sciences, 366 (2008) 3813-3821.

[81]

K. Zyskowski, M.R. Morris, J.P. Bigham, M.L. Gray, S.K. Kane, Accessible crowdwork? Understanding the value in and challenge of microtask employment for people with disabilities, 2015.

Cited By

Boone ARothschild AKoo XPfohl GSheehan ADiSalvo BLe Dantec CDiSalvo C(2024)Reimagining Meaningful Data Work through Citizen ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36870498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687049
Xie HChecco AZamani E(2024)The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurkInformation Systems Frontiers10.1007/s10796-023-10373-x26:1(159-175)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s10796-023-10373-x
Zhang RFlathmann CMusick GSchelble BMcNeese NKnijnenburg BDuan W(2023)I Know This Looks Bad, But I Can Explain: Understanding When AI Should Explain Actions In Human-AI TeamsACM Transactions on Interactive Intelligent Systems10.1145/363547414:1(1-23)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3635474
Show More Cited By

Recommendations

An Analysis of the Use of Qualifications on the Amazon Mechanical Turk Online Labor Market

Several human computation systems use crowdsourcing labor markets to recruit workers. However, it is still a challenge to guarantee that the results produced by workers have a high enough quality. This is particularly difficult in markets based on micro-...
A Social Capital Perspective on IT Professionals' Work Behavior and Attitude

Attracting and retaining information technology IT professionals is a current concern for companies. Although research has been conducted about the job behavior and attitudes of IT professionals over the past three decades, little research has explored ...
Job-Leisure and Work-Family Conflict: Do they Really Matter in IT Professional Turnover?
SIGMIS-CPR '20: Proceedings of the 2020 Computers and People Research Conference

Motivation, job satisfaction, and employee turnover intentions are not new topics in the employment of Information Technology (IT) professionals. Several previous turnover intention studies highlighted already proven factors related to turnover intention ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers in Human Behavior

Computers in Human Behavior Volume 54, Issue C

January 2016

701 pages

ISSN:0747-5632

Issue’s Table of Contents

Copyright © Elsevier Ltd.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 January 2016

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Boone ARothschild AKoo XPfohl GSheehan ADiSalvo BLe Dantec CDiSalvo C(2024)Reimagining Meaningful Data Work through Citizen ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36870498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687049
Xie HChecco AZamani E(2024)The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurkInformation Systems Frontiers10.1007/s10796-023-10373-x26:1(159-175)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s10796-023-10373-x
Zhang RFlathmann CMusick GSchelble BMcNeese NKnijnenburg BDuan W(2023)I Know This Looks Bad, But I Can Explain: Understanding When AI Should Explain Actions In Human-AI TeamsACM Transactions on Interactive Intelligent Systems10.1145/363547414:1(1-23)Online publication date: 2-Dec-2023
https://dl.acm.org/doi/10.1145/3635474
Maddalena EIbáñez LReeves NSimperl E(2023)Qrowdsmith: Enhancing Paid Microtask Crowdsourcing with Gamification and Furtherance IncentivesACM Transactions on Intelligent Systems and Technology10.1145/360494014:5(1-26)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3604940
Kapania STaylor AWang D(2023)A hunt for the Snark: Annotator Diversity in Data PracticesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580645(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580645
Hellwig PBuchholz VKopp SMaier G(2023)Let the user have a say - voice in automated decision-makingComputers in Human Behavior10.1016/j.chb.2022.107446138:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.chb.2022.107446
Song YHu BXue H(2022)Evolution of employee opinion in a crowdsourcing logistics companySimulation10.1177/0037549721106126998:4(347-360)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1177/00375497211061269
Rechkemmer AYin M(2022)Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative ViewProceedings of the ACM on Human-Computer Interaction10.1145/35551376:CSCW2(1-30)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555137
Hettiachchi DKostakos VGoncalves J(2022)A Survey on Task Assignment in CrowdsourcingACM Computing Surveys10.1145/349452255:3(1-35)Online publication date: 3-Feb-2022
https://dl.acm.org/doi/10.1145/3494522
Rothschild ABooker JDavoll CHill JIvey VDiSalvo CRydal Shapiro BDiSalvo B(2022)Towards fair and pro-social employment of digital pieceworkers for sourcing machine learning training dataExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3516384(1-9)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3516384
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents