note

TurkBench: Rendering the Market for Turkers

Authors:

Benjamin V. Hanrahan,

Jutta K. Willamowski,

Saiganesh Swaminathan,

David B. MartinAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Pages 1613 - 1616

https://doi.org/10.1145/2702123.2702279

Published: 18 April 2015 Publication History

Get Access

Abstract

Crowdsourcing is a relatively new model of labor where both the workers and work providers are experiencing its growing pains. A dominant platform that implements this model of labor is Amazon Mechanical Turk (AMT). While AMT has evolved over the years, the changes have focused mainly on work providers and have not addressed the problems workers face (e.g. dealing with market volatility and unpaid time searching for work). In this paper we present emph{TurkBench}, a tool meant to provide workers with personalized market visualization and session management. We discuss the design philosophy of the tool, briefly discuss four Turkers' reaction to a demo, and outline future work.

References

[1]

Bederson, B. B., and Quinn, A. J. Web workers unite! addressing challenges of online laborers. In CHI 2011 EA, ACM (2011), 97--106.

Digital Library

Google Scholar

[2]

Callison-Burch, C. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In HCOMP-2014 (November 2014).

Google Scholar

[3]

Chilton, L. B., Horton, J. J., Miller, R. C., and Azenkot, S. Task search in a human computation market. In ACM SIGKDD Workshop on Human Computation, ACM (2010), 1--9.

Digital Library

Google Scholar

[4]

Ipeirotis, P. G. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads 17, 2 (2010), 16--21.

Digital Library

Google Scholar

[5]

Irani, L. C., and Silberman, M. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In CHI 2013, ACM (2013), 611--620.

Digital Library

Google Scholar

[6]

Khanna, S., Ratan, A., Davis, J., and Thies, W. Evaluating and improving the usability of mechanical turk for low-income workers in india. In DEV '10, ACM (2010).

Digital Library

Google Scholar

[7]

Kittur, A., Khamkar, S., André, P., and Kraut, R. Crowdweaver: Visually managing complex crowd work. In CSCW 2012, ACM (2012), 1033--1036.

Digital Library

Google Scholar

[8]

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. The future of crowd work. In CSCW 2013, ACM (2013), 1301--1318.

Digital Library

Google Scholar

[9]

Kulkarni, A., Can, M., and Hartmann, B. Collaboratively crowdsourcing workflows with turkomatic. In CSCW 2012, ACM (2012), 1003--1012.

Digital Library

Google Scholar

[10]

Martin, D., Hanrahan, B. V., O'Neill, J., and Gupta, N. Being a turker. In CSCW 2014, ACM (2014), pp. 224--235.

Digital Library

Google Scholar

[11]

ONeill, J., and Martin, D. Relationship-based business process crowdsourcing. In INTERACT '13. Springer Berlin Heidelberg, 2013, pp. 429--446.

Google Scholar

[12]

Quinn, A. J., and Bederson, B. B. Human computation: a survey and taxonomy of a growing field. In CHI 2011, ACM (2011), pp. 1403--1412.

Digital Library

Google Scholar

[13]

Silberman, M., Ross, J., Irani, L., and Tomlinson, B. Sellers' problems in human computation markets. In SIGKDD Workshop on Human Computation, ACM (2010), 18--21.

Digital Library

Google Scholar

Cited By

View all

Jaimes Rodriguez IPetterson ADoggett OChandra P(2024)El costo de la independencia: Latino house-cleaners in Technology-Mediated Labour MarketsProceedings of the ACM on Human-Computer Interaction10.1145/36869998:CSCW2(1-32)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686999
Doggett OGram SJit SHarb BHarb HShafik MSoden RRatto M(2024)Digital Gig Work Under Unstable Energy Infrastructures: Invisible Workarounds and Alternative ImaginariesProceedings of the ACM on Human-Computer Interaction10.1145/36869498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686949
Kingsley SSilberman MWang CLambeth RZhi JEslami MLi BBigham J(2024)‘Your Duties Are To Sweep A Floor Remotely’: Low Information Quality in Job Advertisements is a Barrier to Low-Income Job-Seekers’ Successful Use of Digital PlatformsProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663403(1-20)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663403
Show More Cited By

Index Terms

TurkBench: Rendering the Market for Turkers
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Being a turker
CSCW '14: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing

Crowdsourcing is a key current topic in CSCW. We build upon findings of a few qualitative studies of crowdworkers. We conducted an ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (...
A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems

This paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...
How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Note

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
392
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Jaimes Rodriguez IPetterson ADoggett OChandra P(2024)El costo de la independencia: Latino house-cleaners in Technology-Mediated Labour MarketsProceedings of the ACM on Human-Computer Interaction10.1145/36869998:CSCW2(1-32)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686999
Doggett OGram SJit SHarb BHarb HShafik MSoden RRatto M(2024)Digital Gig Work Under Unstable Energy Infrastructures: Invisible Workarounds and Alternative ImaginariesProceedings of the ACM on Human-Computer Interaction10.1145/36869498:CSCW2(1-26)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686949
Kingsley SSilberman MWang CLambeth RZhi JEslami MLi BBigham J(2024)‘Your Duties Are To Sweep A Floor Remotely’: Low Information Quality in Job Advertisements is a Barrier to Low-Income Job-Seekers’ Successful Use of Digital PlatformsProceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3663384.3663403(1-20)Online publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1145/3663384.3663403
Wang JYang YWang SChen CWang DWang Q(2022)Context-Aware Personalized Crowdtesting Task RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2021.308117148:8(3131-3144)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TSE.2021.3081171
Wang QChen ZWang JFeng YWang QChen ZWang JFeng Y(2022)Task Recommendation for Crowd WorkerIntelligent Crowdsourced Testing10.1007/978-981-16-9643-5_5(37-51)Online publication date: 25-Apr-2022
https://doi.org/10.1007/978-981-16-9643-5_5
Sehrawat Usawhney nYeleswarapu TRangaswamy N(2021)The Everyday HCI of Uber Drivers in India: A Developing Country PerspectiveProceedings of the ACM on Human-Computer Interaction10.1145/34795685:CSCW2(1-22)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479568
Rivera VLee D(2021)I Want to, but First I Need toProceedings of the ACM on Human-Computer Interaction10.1145/34492245:CSCW1(1-22)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449224
Hanrahan BChen AMa JMa NSquicciarini ASavage S(2021)The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical TurkProceedings of the ACM on Human-Computer Interaction10.1145/34492025:CSCW1(1-23)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449202
Saito SNakano TKobayashi TBigham JGilbert EKarahalios K(2019)MicroLapseCompanion Publication of the 2019 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3311957.3359466(352-356)Online publication date: 9-Nov-2019
https://dl.acm.org/doi/10.1145/3311957.3359466
Saito SChiang CSavage SNakano TKobayashi TBigham J(2019)TurkScanner: Predicting the Hourly Wage of MicrotasksThe World Wide Web Conference10.1145/3308558.3313716(3187-3193)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313716
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Being a turker

A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters

How many crowdsourced workers should a requester hire?

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations