research-article

Public Access

Crowdsourcing Similarity Judgments for Agreement Analysis in End-User Elicitation Studies

Authors:

Abdullah X. Ali,

Meredith Ringel Morris,

Jacob O. WobbrockAuthors Info & Claims

UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Pages 177 - 188

https://doi.org/10.1145/3242587.3242621

Published: 11 October 2018 Publication History

Abstract

End-user elicitation studies are a popular design method, but their data require substantial time and effort to analyze. In this paper, we present Crowdsensus, a crowd-powered tool that enables researchers to efficiently analyze the results of elicitation studies using subjective human judgment and automatic clustering algorithms. In addition to our own analysis, we asked six expert researchers with experience running and analyzing elicitation studies to analyze an end-user elicitation dataset of 10 functions for operating a web-browser, each with 43 voice commands elicited from end-users for a total of 430 voice commands. We used Crowdsensus to gather similarity judgments of these same 430 commands from 410 online crowd workers. The crowd outperformed the experts by arriving at the same results for seven of eight functions and resolving a function where the experts failed to agree. Also, using Crowdsensus was about four times faster than using experts.

Supplementary Material

suppl.mov (ufp1218p.mp4)

Supplemental video

Download
1.32 MB

MP4 File (p177-ali.mp4)

Download
205.97 MB

References

[1]

Ailon, N., Charikar, M. and Newman, A. (2005). Aggregating inconsistent information: Ranking and clustering. Proceedings of the ACM Symposium on Theory of Computing. New York: ACM Press, pp. 684--693.

Digital Library

[2]

Ailon, N., Charikar, M. and Newman, A. (2008). Aggregating inconsistent information: Ranking and clustering. Journal of the ACM 55 (5), pp. 23:1--23:27.

Digital Library

[3]

Bansal, N., Blum, A. and Chawla, S. (2004). Correlation clustering. Machine Learning 56 (1--3), pp. 89--113.

Digital Library

[4]

Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D. and Panovich, K. (2010). Soylent: a word processor with a crowd inside. Proceedings of the ACM symposium on User interface software and technology. New York: ACM Press, pp. 313--322.

Digital Library

[5]

Bragg, J. and Weld, D.S. (2013) Crowdsourcing multi-label classification for taxonomy creation. Proceedings of AAAI Conference on Human Computation and Crowdsourcing. Palm Springs, CA: HCOMP, pp. 25--33.

[6]

Chan, J., Dang, S. and Dow, S.P. (2016). Improving crowd innovation with expert facilitation. Proceedings of the ACM Conference on Computer-Supported Cooperative Work & Social Computing. New York: ACM Press, pp. 1223--1235.

Digital Library

[7]

Chilton, L.B., Little, G., Edge, D., Weld, D.S. and Landay, J.A. (2013). Cascade: Crowdsourcing taxonomy creation. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1999--2008.

Digital Library

[8]

Demaine, E.D., Emanuel, D., Fiat, A. and Immorlica, N. (2006). Correlation clustering in general weighted graphs. Theoretical Computer Science 361 (2--3), pp. 172--187.

Digital Library

[9]

Findlater, L., Lee, B. and Wobbrock, J.O. (2012). Beyond QWERTY: Augmenting touch screen keyboards with multi-touch gestures for non-alphanumeric input. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 2679--2682.

Digital Library

[10]

Good, M.D., Whiteside, J.A., Wixon, D.R. and Jones, S.J. (1984). Building a user-derived interface. Communications of the ACM 27 (10), pp. 1032-1043.

Digital Library

[11]

Greenhouse, S.W. and Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika 24 (2), pp. 95--112.

[12]

Guo, S., Parameswaran, A., and Garcia-Molina, H. (2012). So who won?: dynamic max discovery with the crowd. Proceedings of the ACM International Conference on Management of Data. New York: ACM Press, pp. 385--396.

Digital Library

[13]

Hoffer, E. and Ailon, N. (2015). Deep metric learning using triplet network. Proceedings of the International Workshop on Similarity-Based Pattern Recognition. Cham: Springer, pp. 84--92.

[14]

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), pp. 65--70.

[15]

Hou, W., Chen, K., Li, H., Zhou, H. (2018). User defined eye movement-based interaction for virtual reality. Proceedings of the International Conference on Cross-Cultural Design. Cham: Springer, pp. 18--30.

[16]

https://math.stackexchange.com/questions/507742/distance-similarity-between-two-matrices:

[17]

Kane, S.K., Wobbrock, J.O. and Ladner, R.E. (2011). Usable gestures for blind people: Understanding preference and performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 413--422.

Digital Library

[18]

Kirkpatrick, S., Gelatt, C.D. and Vecchi, M.P. (1983). Optimization by simulated annealing. Science 220 (4598), pp. 671--680.

[19]

Kittur, A., Smus, B., Khamkar, S. and Kraut, R.E. (2011). Crowdforge: Crowdsourcing complex work. Proceedings of the ACM Symposium on User interface Software and Technology. New York: ACM Press, pp. 43--52.

Digital Library

[20]

Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A. and Möller, S. (2011). I'm home: Defining and evaluating a gesture set for smart-home control. International Journal of Human-Computer Studies 69 (11), pp. 693--704.

[21]

Morris, M.R., Danielescu, A., Drucker, S., Fisher, D., Lee, B., Schraefel, M.C. and Wobbrock, J.O. (2014). Reducing legacy bias in gesture elicitation studies. ACM Interactions 21 (3), pp. 40--45.

Digital Library

[22]

Morris, M.R., Wobbrock, J.O. and Wilson, A.D. (2010). Understanding users' preferences for surface gestures. Proceedings of Graphics Interface. Toronto: Canadian Information Processing Society, pp. 261--268.

Digital Library

[23]

Morris, M.R. (2012). Web on the wall: insights from a multimodal interaction elicitation study. Proceedings of the ACM Conference on Interactive Tabletops and Surfaces. New York: ACM Press, pp. 95--104.

Digital Library

[24]

Nebeling, M., Ott, D., & Norrie, M. C. (2015). Kinect Analysis: A system for recording, analysing and sharing multimodal interaction elicitation studies. Proceedings of SIGCHI Symposium on Engineering Interactive Computing Systems. New York: ACM Press, pp. 142--151.

Digital Library

[25]

Obaid, M., Häring, M., Kistler, F., Buhling, R. and André, E. (2012). User-defined body gestures for navigational control of a humanoid robot. Proceedings of the International Conference on Social Robotics. Berlin: Springer, pp. 367--377.

Digital Library

[26]

Piper, A.M., Campbell, R., and Hollan, J.D. (2010). Exploring the accessibility and appeal of surface computing for older adult health care support. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 907--916.

Digital Library

[27]

Piumsomboon, T., Clark, A., Billinghurst, M. and Cockburn, A. (2013). User-defined gestures for augmented reality. Proceedings of INTERACT 2013. Berlin: Springer, pp. 282-299.

Digital Library

[28]

Vedantam, R., Zitnick, C. L., and Parikh, D. (2015). Cider: Consensus-based image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition. Pp. 4566--4575.

[29]

Stuart J. Russell and Peter Norvig. (1995). Genetic algorithms and evolutionary programming. In Artificial Intelligence: A Modern Approach. Upper Saddle River, NJ: Prentice-Hall, pp. 619--621.

Digital Library

[30]

Stuart J. Russell and Peter Norvig. (1995). Iterative improvement algorithms. In Artificial Intelligence: A Modern Approach. Upper Saddle River, NJ: Prentice-Hall, pp. 111--114.

Digital Library

[31]

Silberman, M.S., B. Tomlinson, R. LaPlante, J. Ross, L. Irani, and A. Zaldivar. (2018). Responsible research with crowds. Communications of the ACM. 61 (3), pp. 39--41.

Digital Library

[32]

Speicher, M. and Nebeling, M. (2018). GestureWiz: A human-powered gesture design environment for user interface prototypes. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp.1--11.

Digital Library

[33]

Tamuz, O., Liu, C., Belongie, S., Shamir, O., and Kalai. A.T., (2011). Adaptively learning the crowd kernel. Proceedings of the 28th International Conference on International Conference on Machine Learning. Omnipress, USA, pp. 673--680.

Digital Library

[34]

Vatavu, R.D. (2012). User-defined gestures for free-hand TV control. Proceedings of the 10th European Conference on Interactive TV and Video. New York: ACM Press, pp. 45--48.

Digital Library

[35]

Vatavu, R.D. and Wobbrock, J.O. (2016). Between-subjects elicitation studies: Formalization and tool support. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 3390--3402.

Digital Library

[36]

Vatavu, R.D. and Wobbrock, J.O. (2015). Formalizing agreement analysis for elicitation studies: New measures, significance test, and toolkit. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1325--1334.

Digital Library

[37]

Wobbrock, J.O., Aung, H.H., Rothrock, B. and Myers, B.A. (2005). Maximizing the guessability of symbolic input. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1869--1872.

Digital Library

[38]

Wobbrock, J.O., Morris, M.R. and Wilson, A.D. (2009). User-defined gestures for surface computing. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM Press, pp. 1083--1092.

Digital Library

[39]

Zhang, Z., Cheng, H., Chen, W., Zhang, S. and Fang, Q. (2008). Correlation clustering based on genetic algorithm for documents clustering. Proceedings of the IEEE Congress on Evolutionary Computation. Piscataway, NJ: IEEE Press, pp. 3193--3198.

Cited By

Friedman ACafaro F(2024)Designing Gestures for Data Exploration with Public Displays via Identification StudiesInformation10.3390/info1506029215:6(292)Online publication date: 21-May-2024
https://doi.org/10.3390/info15060292
Datey IZytko D(2024)"Just Like, Risking Your Life Here": Participatory Design of User Interactions with Risk Detection AI to Prevent Online-to-Offline Harm Through Dating AppsProceedings of the ACM on Human-Computer Interaction10.1145/36869068:CSCW2(1-41)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686906
Cheng YWu ZXiao R(2024)Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.338726912(64958-64979)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387269
Show More Cited By

Index Terms

Crowdsourcing Similarity Judgments for Agreement Analysis in End-User Elicitation Studies
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Crowdlicit: A System for Conducting Distributed End-User Elicitation and Identification Studies
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

End-user elicitation studies are a popular design method. Currently, such studies are usually confined to a lab, limiting the number and diversity of participants, and therefore the representativeness of their results. Furthermore, the quality of the ...
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous ...
Pricing mechanisms for crowdsourcing markets
WWW '13: Proceedings of the 22nd international conference on World Wide Web

Every day millions of crowdsourcing tasks are performed in exchange for payments. Despite the important role pricing plays in crowdsourcing campaigns and the complexity of the market, most platforms do not provide requesters appropriate tools for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

October 2018

1016 pages

ISBN:9781450359481

DOI:10.1145/3242587

General Chairs:
Patrick Baudisch
Hasso-Plattner Institute, Germany
,
Albrecht Schmidt
LMU, Germany
,
Program Chair:
Andy Wilson
Microsoft Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

UIST '18

Sponsor:

UIST '18: The 31st Annual ACM Symposium on User Interface Software and Technology

October 14, 2018

Berlin, Germany

Acceptance Rates

UIST '18 Paper Acceptance Rate 80 of 375 submissions, 21%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
680
Total Downloads

Downloads (Last 12 months)121
Downloads (Last 6 weeks)13

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Friedman ACafaro F(2024)Designing Gestures for Data Exploration with Public Displays via Identification StudiesInformation10.3390/info1506029215:6(292)Online publication date: 21-May-2024
https://doi.org/10.3390/info15060292
Datey IZytko D(2024)"Just Like, Risking Your Life Here": Participatory Design of User Interactions with Risk Detection AI to Prevent Online-to-Offline Harm Through Dating AppsProceedings of the ACM on Human-Computer Interaction10.1145/36869068:CSCW2(1-41)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686906
Cheng YWu ZXiao R(2024)Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.338726912(64958-64979)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387269
Villarreal-Narvaez SSluÿters AVanderdonckt JVatavu R(2023)Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation StudiesACM Computing Surveys10.1145/363645856:5(1-55)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3636458
Jensen JAshbrook D(2023)Exploring Audio Icons for Content-Based Navigation in Voice User InterfacesProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3604302(1-9)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3604302
Hosseini MIhmels TChen ZKoelle MMüller HBoll S(2023)Towards a Consensus Gesture Set: A Survey of Mid-Air Gestures in HCI for Maximized Agreement Across DomainsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581420(1-24)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581420
Jannat MYang XHasan K(2023)Around-device finger input on commodity smartwatches with learning guidance through discoverabilityInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103105179:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103105
Vatavu R(2023)Gesture-Based InteractionHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_20-1(1-47)Online publication date: 9-Feb-2023
https://doi.org/10.1007/978-3-319-27648-9_20-1
Xia HGlueck MAnnett MWang MWigdor D(2022)Iteratively Designing Gesture Vocabularies: A Survey and Analysis of Best Practices in the HCI LiteratureACM Transactions on Computer-Human Interaction10.1145/350353729:4(1-54)Online publication date: 5-May-2022
https://dl.acm.org/doi/10.1145/3503537
Zhang MZhong MWobbrock J(2022)Ga11y: An Automated GIF Annotation System for Visually Impaired UsersProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502092(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502092
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents