research-article

Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal

Authors:

Judith Alice Redi,

Tobias Hoßfeld,

Pavel Korshunov,

Christian KeimelAuthors Info & Claims

CrowdMM '13: Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia

Pages 29 - 34

https://doi.org/10.1145/2506364.2506368

Published: 22 October 2013 Publication History

Abstract

Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.

References

[1]

O. Axelsson. Towards a psychology of photography: Dimensions underlying aesthetic appeal of photographs. Perceptual and Motor Skills, 105(August 2002):411--434, 2007.

[2]

K.-T. Chen, C.-J. Chang, C.-C. Wu, Y.-C. Chang, and C.-L. Lei. Quadrant of euphoria: a crowdsourcing platform for QoE assessment. Network, IEEE, 24(2):28--35, Mar. 2010.

Digital Library

[3]

R. Datta, D. Joshi, J. Li, and J. Z. Wang. Studying aesthetics in photographic images using a computational approach. In Proceedings of the 9th European conference on Computer Vision - Volume Part III, ECCV'06, pages 288--301, Berlin, Heidelberg, 2006. Springer-Verlag.

Digital Library

[4]

A. Doan, R. Ramakrishnan, and A. Y. Halevy. Crowdsourcing systems on the world-wide web. Commun. ACM, 54(4):86--96, Apr. 2011.

Digital Library

[5]

M. Hirth, T. Hoßfeld, and P. Tran-Gia. Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com. In Workshop on Future Internet and Next Generation Networks (FINGNet), pages 322--329, Seoul, Korea, June 2011.

Digital Library

[6]

T. Hoßfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia. CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation. Technical Report 486, University of Würzburg, Feb. 2013.

[7]

T. Hoßfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz. Quantification of youtube qoe via crowdsourcing. In Multimedia (ISM), 2011 IEEE International Symposium on, pages 494--499, Dec. 2011.

Digital Library

[8]

P. Isola, J. Xiao, A. Torralba, and A. Oliva. What makes an image memorable? In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 145--152. IEEE, 2011.

Digital Library

[9]

D. Joshi, R. Datta, E. Fedorovskaya, Q.-T. Luong, J. Wang, J. Li, and J. Luo. Aesthetics and emotions in images. Signal Processing Magazine, IEEE, 28(5):94--115, 2011.

[10]

D. Joshi, R. Datta, E. Fedorovskaya, Q.-T. Luong, J. Z. Wang, J. Li, and J. Luo. Aesthetics and emotions in images. Signal Processing Magazine, IEEE, 28(5):94--115, 2011.

[11]

C. Keimel, J. Habigt, C. Horch, and K. Diepold. Qualitycrowd - a framework for crowd-based quality evaluation. In Picture Coding Symposium (PCS), 2012, pages 245--248, May 2012.

[12]

A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 453--456, New York, NY, USA, 2008. ACM.

Digital Library

[13]

P. Korshunov, S. Cai, and T. Ebrahimi. Crowdsourcing approach for evaluation of privacy filters in video surveillance. In Proceedings of the ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia, CrowdMM'12, pages 35--40, Nara, Japan, Oct. 2012.

Digital Library

[14]

J. Lassalle, S. Member, L. Gros, T. Morineau, and G. Coppin. Impact of the content on subjective evaluation of audiovisual quality: What dimensions influence our perception? In IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pages 1--6, 2012.

[15]

P. Le Callet, S. Möller, and A. Perkis. Qualinet white paper on definitions of quality of experience (2012). European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003), Mar. 2013.

[16]

C. Li and T. Chen. Aesthetic visual quality assessment of paintings. Selected Topics in Signal Processing, IEEE journal, 3(2):236--252, 2009.

[17]

W. A. Mansilla, A. Perkis, and T. Ebrahimi. Implicit experiences as a determinant of perceptual quality and aesthetic appreciation. In Proceedings of the 19th ACM international conference on Multimedia, pages 153--162. ACM, 2011.

Digital Library

[18]

S. Plous. The Psychology of Judgment and Decision Making. McGraw-Hill Series in Social Psychology. McGraw-Hill Education, 1993.

[19]

Recommendation. ITU-R BT.500--13. Methodology for the subjective assessment of the quality of television pictures. International Telecommunication Union, Geneva, Switzerland, 2012.

[20]

J. A. Redi. Visual quality beyond artifact visibility. Proc. SPIE, 8651:86510N-86510N--11, Mar. 2013.

[21]

S. Suri, D. Goldstein, and W. Mason. Honesty in an online labor market. In Human Computation: Papers from the 2011 AAAI Workshop, pages 61--66, 2011.

[22]

W. Tschacher, S. Greenwood, V. Kirchberg, S. Wintzerith, K. van den Berg, and M. Tröndle. Physiological correlates of aesthetic perception of artworks in a museum. Psychology of Aesthetics, Creativity, and the Arts, 6(1):96--103, 2012.

[23]

L.-k. Wong and K.-l. Low. Saliency-enhanced image aesthetics class prediction. In 2009 16th IEEE International Conference on Image Processing (ICIP), pages 997--1000, Cairo, Nov. 2009. Ieee.

Digital Library

Cited By

Sharma KSherk TPatel VTran MNguyen T(2025)GAIA: A Benchmark of Analyzing User Rankings for Synthesized ImagesAdvances in Visual Computing10.1007/978-3-031-77392-1_34(451-463)Online publication date: 22-Jan-2025
https://doi.org/10.1007/978-3-031-77392-1_34
Alves SCosta RMontague KGuerreiro T(2024)Citizen-Led Personalization of User Interfaces: Investigating How People Customize Interfaces for Themselves and OthersProceedings of the ACM on Human-Computer Interaction10.1145/36869858:CSCW2(1-23)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686985
Schwarze SSchaadt NSobotta VSpicher NSkripuletz TEsmaeilzadeh MKrauss JHartmann CDeserno TFeuerhake F(2024)Task design for crowdsourced glioma cell annotation in microscopy imagesScientific Reports10.1038/s41598-024-51995-814:1Online publication date: 23-Jan-2024
https://doi.org/10.1038/s41598-024-51995-8
Show More Cited By

Index Terms

Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Graphics systems and interfaces
      1. Perception

Recommendations

Crowdsourcing for Rating Image Aesthetic Appeal: Better a Paid or a Volunteer Crowd?
CrowdMM '14: Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia

Crowdsourcing has the potential to become a preferred tool to study image aesthetic appeal preferences of users. Nevertheless, some reliability issues still exist, partially due to the sometimes doubtful commitment of paid workers to perform the rating ...
Assessing internet video quality using crowdsourcing
CrowdMM '13: Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia

In this paper, we present a subjective video quality evaluation system that has been integrated with different crowdsourcing platforms. We try to evaluate the feasibility of replacing the time consuming and expensive traditional tests with a faster and ...
Intra- and Inter-rater Agreement in a Subjective Speech Quality Assessment Task in Crowdsourcing
WWW '19: Companion Proceedings of The 2019 World Wide Web Conference

Crowdsourcing is a great tool for conducting subjective user studies with large amounts of users. Collecting reliable annotations about the quality of speech stimuli is challenging. The task itself is of high subjectivity and users in crowdsourcing work ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CrowdMM '13: Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia

October 2013

44 pages

ISBN:9781450323963

DOI:10.1145/2506364

General Chairs:
Kuan-Ta Chen
Academia Sinica, Taiwan
,
Wei-Ta Chu
National Chung Cheng University, Taiwan
,
Martha Larson
Delft University of Technology, The Netherlands

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '13

Sponsor:

SIGMM

MM '13: ACM Multimedia Conference

October 22, 2013

Barcelona, Spain

Acceptance Rates

CrowdMM '13 Paper Acceptance Rate 8 of 16 submissions, 50%;

Overall Acceptance Rate 16 of 42 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
535
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sharma KSherk TPatel VTran MNguyen T(2025)GAIA: A Benchmark of Analyzing User Rankings for Synthesized ImagesAdvances in Visual Computing10.1007/978-3-031-77392-1_34(451-463)Online publication date: 22-Jan-2025
https://doi.org/10.1007/978-3-031-77392-1_34
Alves SCosta RMontague KGuerreiro T(2024)Citizen-Led Personalization of User Interfaces: Investigating How People Customize Interfaces for Themselves and OthersProceedings of the ACM on Human-Computer Interaction10.1145/36869858:CSCW2(1-23)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686985
Schwarze SSchaadt NSobotta VSpicher NSkripuletz TEsmaeilzadeh MKrauss JHartmann CDeserno TFeuerhake F(2024)Task design for crowdsourced glioma cell annotation in microscopy imagesScientific Reports10.1038/s41598-024-51995-814:1Online publication date: 23-Jan-2024
https://doi.org/10.1038/s41598-024-51995-8
Chen SVaxman YBaruch EAsulin DMoreshet ASra MSen P(2024)AID-AppEAL: Automatic Image Dataset and Algorithm for Content Appeal Enhancement and Assessment LabelingComputer Vision – ECCV 202410.1007/978-3-031-72655-2_2(19-36)Online publication date: 6-Dec-2024
https://doi.org/10.1007/978-3-031-72655-2_2
Zhang TXia CLiu ZZhao SPeng HYu P(2023)Domain-Invariant Feature Progressive Distillation with Adversarial Adaptive Augmentation for Low-Resource Cross-Domain NERACM Transactions on Asian and Low-Resource Language Information Processing10.1145/357050222:3(1-21)Online publication date: 14-Apr-2023
https://dl.acm.org/doi/10.1145/3570502
Lu YNath SPentapati SLim S(2023)ECO-GNN: Signoff Power Prediction Using Graph Neural Networks with Subgraph ApproximationACM Transactions on Design Automation of Electronic Systems10.1145/356994228:4(1-22)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1145/3569942
Zhang QHuang HLi JZhang YLi Y(2023)CmpCNN: CMP Modeling with Transfer Learning CNN ArchitectureACM Transactions on Design Automation of Electronic Systems10.1145/356994128:4(1-18)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1145/3569941
Ma TZhang ZRong HAl-Nabhan N(2023)SPK-CG: Siamese Network based Posterior Knowledge Selection Model for Knowledge Driven Conversation GenerationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/356957922:3(1-16)Online publication date: 10-Mar-2023
https://dl.acm.org/doi/10.1145/3569579
Hou TZhen PJi ZChen H(2023)A Deep Learning Framework for Solving Stress-based Partial Differential Equations in Electromigration AnalysisACM Transactions on Design Automation of Electronic Systems10.1145/356742428:4(1-20)Online publication date: 17-May-2023
https://dl.acm.org/doi/10.1145/3567424
Bommana ASiddamshetty SPudi DThumatti K. R. ABoppu SSabarimalai Manikandan MCenkeramaddi L(2023)Design of Synthesis-time Vectorized Arithmetic Hardware for Tapered Floating-point Addition and SubtractionACM Transactions on Design Automation of Electronic Systems10.1145/356742328:3(1-35)Online publication date: 22-Mar-2023
https://dl.acm.org/doi/10.1145/3567423
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten