poster

Worker types and personality traits in crowdsourcing relevance labels

Authors:

Gabriella Kazai,

Natasa Milic-FraylingAuthors Info & Claims

CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

Pages 1941 - 1944

https://doi.org/10.1145/2063576.2063860

Published: 24 October 2011 Publication History

Abstract

Crowdsourcing platforms offer unprecedented opportunities for creating evaluation benchmarks, but suffer from varied output quality from crowd workers who possess different levels of competence and aspiration. This raises new challenges for quality control and requires an in-depth understanding of how workers' characteristics relate to the quality of their work.

In this paper, we use behavioral observations (HIT completion time, fraction of useful labels, label accuracy) to define five worker types: Spammer, Sloppy, Incompetent, Competent, Diligent. Using data collected from workers engaged in the crowdsourced evaluation of the INEX 2010 Book Track Prove It task, we relate the worker types to label accuracy and personality trait information along the `Big Five' personality dimensions.

We expect that these new insights about the types of crowd workers and the quality of their work will inform how to design HITs to attract the best workers to a task and explain why certain HIT designs are more effective than others.

References

[1]

O. Alonso and R. A. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Proc. ECIR'11, pages 153--164, 2011.

Digital Library

[2]

O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42: 9--15, November 2008.

Digital Library

[3]

B. Carterette and I. Soboroff. The effect of assessor error on ir system evaluation. In Proc. SIGIR'10, pages 539--546. ACM, 2010.

Digital Library

[4]

J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening Mechanical Turk workers. In Proc. CHI'10, pages 2399--2402, 2010.

Digital Library

[5]

S. D. Gosling, S. Gaddis, and S. Vazire. Personality impressions based on Facebook profiles. Psychology, pages 1--4, 2007.

[6]

C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In Proc. CSLDAMT'10, pages 172--179, 2010.

Digital Library

[7]

J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, 2008.

Digital Library

[8]

J.-H. Huang and Y.-C. Yang. The relationship between personality traits and online shopping motivations. Social Behavior and Personality, 38: 673--680, 2010.

[9]

P. G. Ipeirotis. Analyzing the Amazon Mechanical Turk marketplace. XRDS, 17: 16--21, 2010.

Digital Library

[10]

O. P. John, L. P. Naumann, and C. J. Soto. Paradigm shift to the integrative big-five trait taxonomy. In Handbook of personality, chapter 4, pages 114--212. Guilford Press, New York NY, 2008.

[11]

G. Kazai. In search of quality in crowdsourcing for search engine evaluation. In Proc. ECIR'11, pages 165--176, 2011.

Digital Library

[12]

Kazai, Kamps, Koolen, and Milic-Frayling}kazai11sigirG. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In Proc. SIGIR'11, pages 205--214, 2011.

Digital Library

[13]

Kazai, Koolen, Kamps, Doucet, and Landoni}kaza:over11G. Kazai, M. Koolen, J. Kamps, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing. In Proc. INEX'10, pages 101--120, 2011.

Digital Library

[14]

A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI'08, CHI '08, pages 453--456, 2008.

Digital Library

[15]

M. Kosinski, F. Radlinski, and P. Kohli. Personality and online behavior. In Proc. CIKM'11, 2011. ACM.

[16]

J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. CSE'10, pages 21--26, 2010.

[17]

B. Rammstedt and O. P. John. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41: 203--212, 2007.

[18]

J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in Mechanical Turk. In Proc. CHI 2010, Extended Abstracts Volume, pages 2863--2872. ACM, 2010.

Digital Library

[19]

R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast--but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. EMNLP'08, pages 254--263, 2008.

Digital Library

[20]

J. Vuurens, A. P. de Vries, and C. Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11), pages 21--26, 2011. ACM.

[21]

D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In Proc. CSE'10, pages 17--20, 2010.

Cited By

Lin FTsai PLee CHo YChen YYen YChang Y(2024)“I Prefer Regular Visitors to Answer My Questions”: Users’ Desired Experiential Background of Contributors for Location-based Crowdsourcing PlatformProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642520(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642520
Kang XYu GKong LDomeniconi CZhang XLi Q(2024)FedTA: Federated Worthy Task Assignment for Crowd WorkersIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.334618321:4(4098-4109)Online publication date: Jul-2024
https://doi.org/10.1109/TDSC.2023.3346183
Mourelatos E(2024)Mood matters: the interplay of personality in ethical perceptions in crowdsourcingBehaviour & Information Technology10.1080/0144929X.2024.2349786(1-23)Online publication date: 17-May-2024
https://doi.org/10.1080/0144929X.2024.2349786
Show More Cited By

Index Terms

Worker types and personality traits in crowdsourcing relevance labels
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy
CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge management

Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a ...
Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
Abstract
The suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a ...
Swedish IT Project Managers' Personality Traits Mirrored in the Big Five

Project success, which is critical to achieve, requires a competent project manager. Could anybody become a skilled project manager, or what does it take? One factor that is considered to influence the opportunities of succeeding is individuals' ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

October 2011

2712 pages

ISBN:9781450307178

DOI:10.1145/2063576

Editors:
Bettina Berendt,
Arjen de Vries,
Wenfei Fan,
Craig Macdonald
University of Glasgow, UK
,
Iadh Ounis
University of Glasgow, UK
,
Ian Ruthven
University of Strathclyde, UK

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

CIKM '11

Sponsor:

CIKM '11: International Conference on Information and Knowledge Management

October 24 - 28, 2011

Glasgow, Scotland, UK

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

101
Total Citations
View Citations
918
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)9

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin FTsai PLee CHo YChen YYen YChang Y(2024)“I Prefer Regular Visitors to Answer My Questions”: Users’ Desired Experiential Background of Contributors for Location-based Crowdsourcing PlatformProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642520(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642520
Kang XYu GKong LDomeniconi CZhang XLi Q(2024)FedTA: Federated Worthy Task Assignment for Crowd WorkersIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.334618321:4(4098-4109)Online publication date: Jul-2024
https://doi.org/10.1109/TDSC.2023.3346183
Mourelatos E(2024)Mood matters: the interplay of personality in ethical perceptions in crowdsourcingBehaviour & Information Technology10.1080/0144929X.2024.2349786(1-23)Online publication date: 17-May-2024
https://doi.org/10.1080/0144929X.2024.2349786
Hedderich MOulasvirta A(2024)Explaining crowdworker behaviour through computational rationalityBehaviour & Information Technology10.1080/0144929X.2024.2329616(1-22)Online publication date: 24-Apr-2024
https://doi.org/10.1080/0144929X.2024.2329616
Wijenayake SHettiachchi DGoncalves J(2023)Combining Worker Factors for Heterogeneous Crowd Task AssignmentProceedings of the ACM Web Conference 202310.1145/3543507.3583190(3794-3805)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583190
Xiao MJin WLi MYang LThapa ALi P(2023)Collusion-Resistant Worker Recruitment in Crowdsourcing SystemsIEEE Transactions on Mobile Computing10.1109/TMC.2021.307109322:1(129-144)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TMC.2021.3071093
Wang YChen XZhou X(2023)A New Method for Identifying Low-Quality Data in Perceived Usability Crowdsourcing Tests: Differences in Questionnaire ScoresInternational Journal of Human–Computer Interaction10.1080/10447318.2023.226369440:22(7297-7313)Online publication date: 9-Oct-2023
https://doi.org/10.1080/10447318.2023.2263694
Mourelatos EGiannakopoulos NTzagarakis M(2023)Payment schemes in online labour markets. Does incentive and personality matter?Behaviour & Information Technology10.1080/0144929X.2023.2254853(1-22)Online publication date: 11-Sep-2023
https://doi.org/10.1080/0144929X.2023.2254853
Paulino DCorreia ABarroso JParedes H(2023)Cognitive personalization for online microtask labor platforms: A systematic literature reviewUser Modeling and User-Adapted Interaction10.1007/s11257-023-09383-w34:3(617-658)Online publication date: 19-Sep-2023
https://doi.org/10.1007/s11257-023-09383-w
Gilal ATunio MWaqas AAlmomani MKhan SGilal R(2022)Task Assignment and PersonalityResearch Anthology on Agile Software, Software Development, and Testing10.4018/978-1-6684-3702-5.ch086(1795-1809)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3702-5.ch086
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents