note

The Effects of Sequence and Delay on Crowd Work

Authors:

Walter S. Lasecki,

Jeffrey M. Rzeszotarski,

Adam Marcus,

Jeffrey P. BighamAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 1375 - 1378

https://doi.org/10.1145/2702123.2702594

Published: 18 April 2015 Publication History

Get Access

Abstract

A common approach in crowdsourcing is to break large tasks into small microtasks so that they can be parallelized across many crowd workers and so that redundant work can be more easily compared for quality control. In practice, this can result in the microtasks being presented out of their natural order and often introduces delays between individual microtasks. In this paper, we demonstrate in a study of 338 crowd workers that non-sequential microtasks and the introduction of delays significantly decreases worker performance. We show that interruptions where a large delay occurs between two related tasks can cause up to a 102% slowdown in completion time, and interruptions where workers are asked to perform different tasks in sequence can slow down completion time by 57%. We conclude with a set of design guidelines to improve both worker performance and realized pay, and instructions for implementing these changes in existing interfaces for crowd work.

References

[1]

Adamczyk, P. D., and Bailey, B. P. If not now, when?: the effects of interruption at different moments within task execution. In CHI 2004.

Digital Library

Google Scholar

[2]

Bernstein, M. S., et al.Soylent: a word processor with a crowd inside. In Proc. UIST, pages 313--322, 2010.

Digital Library

Google Scholar

[3]

Bigham., J. P., et. al. Vizwiz: nearly real-time answers to visual questions. In Proc. UIST, pages 333--342, 2010.

Digital Library

Google Scholar

[4]

Cabon, P., Coblentz, A., and Mollard, R. Interruption of a monotonous activity with complex tasks: effects of individual differences. In Human Factors Society, 1990.

Google Scholar

[5]

Chilton, L. B., Horton, J. J., Miller, R. C., and Azenkot, S. Task search in a human computation market. In HCOMP 2010.

Digital Library

Google Scholar

[6]

Cutrell, E., Czerwinski, M., and Horvitz, E. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Proc. INTERACT, pages 263--269, 2001.

Google Scholar

[7]

Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. Shepherding the crowd yields better work. In Proc. CSCW, pages 1013--1022, 2012.

Digital Library

Google Scholar

[8]

Gillie, T. and Broadbent, D. What makes interruptions disruptive? a study of length, similarity, and complexity. Psychological Research, 50:243--50, 1989.

Crossref

Google Scholar

[9]

Jansset, C. P., and Brumby, D. P. Strategic adaptation to performance objectives in a dualtask setting. Cognitive Science, 34(8):1548--1560, 2010.

Crossref

Google Scholar

[10]

Kreifeldt, J. G., and McCarthy, M. E. Interruption as a test of the user-computer interface. In Proc. MC, 1981.

Google Scholar

[11]

Lasecki, W. S., Murray, K., White, S., Miller, R. C., and Bigham, J. P. Real-time crowd control of existing interfaces. In Proc. UIST, pages 23--32, 2011.

Digital Library

Google Scholar

[12]

Lasecki, W. S., White, S., Murray, K. I., and Bigham, J. P. Crowd memory: Learning in the collective. In Proc. Collective Intelligence, 2012.

Google Scholar

[13]

Pattyn, N., Neyt, X., Henderickx, D., and Soetens, E. Psychophysiological investigation of vigilance decrement: boredom or cognitive fatigue? Physiology and Behavior, 93(1-2):369--378, 2008.

Crossref

Google Scholar

Cited By

View all

Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Yadav DKarlgren KShaikh RHelms KMcmillan DBrown BLampinen A(2024)Bodywork at Work: Attending to Bodily Needs in Gig, Shift, and Knowledge WorkProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642416(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642416
Lascău LBrumby DGould SCox A(2023)“Sometimes It’s Like Putting the Track in Front of the Rushing Train”: Having to Be ‘On Call’ for Work Limits the Temporal Flexibility of CrowdworkersACM Transactions on Computer-Human Interaction10.1145/363514531:2(1-45)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3635145
Show More Cited By

Index Terms

The Effects of Sequence and Delay on Crowd Work
1. Information systems
  1. Information systems applications

Recommendations

How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Shepherding the crowd: managing and providing feedback to crowd workers
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing Systems

Micro-task platforms provide a marketplace for hiring people to do short-term work for small payments. Requesters often struggle to obtain high-quality results, especially on content-creation tasks, because work cannot be easily verified and workers can ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Note

Funding Sources

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
350
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Salim SHoque MMueller K(2024)Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General PopulationsProceedings of the ACM on Human-Computer Interaction10.1145/36372988:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637298
Yadav DKarlgren KShaikh RHelms KMcmillan DBrown BLampinen A(2024)Bodywork at Work: Attending to Bodily Needs in Gig, Shift, and Knowledge WorkProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642416(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642416
Lascău LBrumby DGould SCox A(2023)“Sometimes It’s Like Putting the Track in Front of the Rushing Train”: Having to Be ‘On Call’ for Work Limits the Temporal Flexibility of CrowdworkersACM Transactions on Computer-Human Interaction10.1145/363514531:2(1-45)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3635145
Kerrigan DHullman JBertini E(2021)A Survey of Domain Knowledge Elicitation in Applied Machine LearningMultimodal Technologies and Interaction10.3390/mti51200735:12(73)Online publication date: 24-Nov-2021
https://doi.org/10.3390/mti5120073
Toxtli CSuri SSavage S(2021)Quantifying the Invisible Labor in Crowd WorkProceedings of the ACM on Human-Computer Interaction10.1145/34760605:CSCW2(1-26)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3476060
Ramesh DLiu AEcheverria ASong JWaytowich NLasecki WEl Fallah Seghrouchni ASukthankar GAn BYorke-Smith N(2020)Yesterday's Reward is Today's PunishmentProceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3398761.3398888(1090-1097)Online publication date: 5-May-2020
https://dl.acm.org/doi/10.5555/3398761.3398888
Rahman PNandi AHebert C(2020)Amplifying Domain Expertise in Clinical Data PipelinesJMIR Medical Informatics10.2196/196128:11(e19612)Online publication date: 5-Nov-2020
https://doi.org/10.2196/19612
Rani UFurrer M(2020)Digital labour platforms and new forms of flexible work in developing countries: Algorithmic management of work and workersCompetition & Change10.1177/102452942090518725:2(212-236)Online publication date: 23-Feb-2020
https://doi.org/10.1177/1024529420905187
Oppenlaender JHosio SPaternò FJacucci GRohs MSantoro C(2019)Design recommendations for augmenting creative tasks with computational primingProceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia10.1145/3365610.3365621(1-13)Online publication date: 26-Nov-2019
https://dl.acm.org/doi/10.1145/3365610.3365621
Lascau LGould SCox AKarmannaya EBrumby DBrewster SFitzpatrick GCox AKostakos V(2019)Monotasking or MultitaskingProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300649(1-14)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300649
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

How many crowdsourced workers should a requester hire?

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

Shepherding the crowd: managing and providing feedback to crowd workers