research-article

Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool

Authors:

Rebecca Krosnick,

Sun Young Park,

Brandon Keelean,

Stephanie D. O'Keefe,

Walter S. LaseckiAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 2, Issue CSCW

Article No.: 104, Pages 1 - 23

https://doi.org/10.1145/3274373

Published: 01 November 2018 Publication History

Abstract

Real-time collaboration between a requester and crowd workers expands the scope of tasks that crowdsourcing can be used for by letting requesters and crowd workers interactively create various artifacts (e.g., a sketch prototype, writing, or program code). In such systems, it is increasingly common to allow requesters to verbally describe their requests, receive responses from workers, and provide immediate and continuous feedback to enhance the overall outcome of the two groups' real-time collaboration. This work is motivated by the lack of a deep understanding of the challenges that end users of such systems face in their communication with workers and the need of design implications that can address such challenges for other similar systems. In this paper, we investigate how requesters verbally communicate and collaborate with crowd workers to solve a complex task. Using a crowd-powered UI design tool, we conducted a qualitative user study to explore how requesters with varying expertise communicate and collaborate with crowd workers. Our work also identifies the unique challenges that collaborative crowdsourcing systems pose: potential expertise differences between requesters and crowd workers, the asymmetry of two-way communication (e.g., speech versus text), and the shared artifact's concurrent modification by two disparate groups. Finally, we make design recommendations that can inform the design of future real-time collaboration processes in crowdsourcing systems.

References

[1]

2017. Amazon's Mechanical Turk. http://www.mturk.com.

[2]

2017. Voice Qualities. http://www.ncvs.org/ncvs/tutorials/voiceprod/tutorial/quality.html.

[3]

Salvatore Andolina, Hendrik Schneider, Joel Chan, Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017. Crowdboard:Augmenting In-Person Idea Generation with Real-Time Crowds. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17). ACM, New York, NY, USA, 106--118.

Digital Library

[4]

Adam Bergkvist, Daniel C Burnett, Cullen Jennings, and Anant Narayanan. 2012. Webrtc 1.0: Real-time communication between browsers. Working draft, W3C 91 (2012).

[5]

Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 33--42.

Digital Library

[6]

Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell,and Katrina Panovich. 2015. Soylent: A Word Processor with a Crowd Inside. Commun. ACM 58, 8 (July 2015), 85--94.

Digital Library

[7]

Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions .In User Interface Software and Technology (UIST). 333--342.

Digital Library

[8]

Yan Chen, Sang Won Lee, Yin Xie, Yiwei Yang, Walter S. Lasecki, and Steve Oney. 2017. Codeon: On-Demand Software Development Assistance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6220--6231.

Digital Library

[9]

Domenic V Cicchetti. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. In Psychological Assessment, 6(4). 284--290.

[10]

Herbert H Clark, Susan E Brennan, et al. 1991. Grounding in communication. Perspectives on socially shared cognition13, 1991 (1991), 127--149.

[11]

Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the Crowd Yields Better Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY,USA, 1013--1022.

Digital Library

[12]

Clarence A. Ellis, Simon J. Gibbs, and Gail Rein. 1991. Groupware: Some Issues and Experiences. Commun. ACM 34, 1(Jan. 1991), 39--58.

Digital Library

[13]

Susan R. Fussell, Robert E. Kraut, and Jane Siegel. 2000. Coordination of Communication: Effects of Shared Visual Context on Collaborative Work. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, New York, NY, USA, 21--30.

Digital Library

[14]

Darren Gergle, Robert E. Kraut, and Susan R. Fussell. 2004. Action As Language in a Shared Visual Space. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 487--496.

Digital Library

[15]

Darren Gergle, Robert E. Kraut, and Susan R. Fussell. 2004. Language Efficiency and Visual Technology: Minimizing Collaborative Effort with Visual Information. Journal of Language and Social Psychology 23, 4 (2004), 491--517. arXiv:https://doi.org/10.1177/0261927X04269589

[16]

Mitchell Gordon, Jeffrey P. Bigham, andWalter S. Lasecki. 2015. LegionTools: A Toolkit + UI for Recruiting and Routing Crowds to Synchronous Real-Time Tasks. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15 Adjunct). ACM, New York, NY, USA, 81--82.

Digital Library

[17]

Sai R. Gouravajhala, Jinyeong Yim, Karthik Desingh, Yanda Huang, Odest Chadwicke Jenkins, and Walter S. Lasecki.2018. EURECA: Enhanced Understanding of Real Environments via Crowd Assistance. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Zürich, Switzerland, July 5--8, 2018. 31--40. https://aaai.org/ocs/index.php/HCOMP/HCOMP18/paper/view/17933

[18]

Saul Greenberg and David Marwood. 1994. Real Time Groupware As a Distributed System: Concurrency Control and Its Effect on the Interface. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW'94). ACM, New York, NY, USA, 207--217.

Digital Library

[19]

Carl Gutwin. 2001. The effects of network delays on group work in real-time groupware. In ECSCW 2001. Springer,299--318.

Digital Library

[20]

Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work (CSCW) 11, 3--4 (2002), 411--446.

Digital Library

[21]

Carl Gutwin and Reagan Penner. 2002. Improving Interpretation of Remote Gestures with Telepointer Traces. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (CSCW '02). ACM, New York, NY,USA, 49--57.

Digital Library

[22]

Oliver L. Haimson and John C. Tang. 2017. What Makes Live Events Engaging on Facebook Live, Periscope, and Snapchat. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 48--60.

Digital Library

[23]

William A. Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on Twitch: Fostering Participatory Communities of Play Within Live Mixed Media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1315--1324.

Digital Library

[24]

Pamela J Hinds. 1999. The curse of expertise: The effects of expertise and debiasing methods on prediction of novice performance. Journal of Experimental Psychology: Applied 5, 2 (1999), 205.

[25]

Pamela J Hinds, Michael Patterson, and Jeffrey Pfeffer. 2001. Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Journal of applied psychology 86, 6 (2001), 1232.

[26]

Simo Hosio, Jorge Goncalves, Niels van Berkel, Simon Klakegg, Shin Ichi Konomi, and Vassilis Kostakos. 2017. Facilitating Collocated Crowdsourcing on Situated Displays. Human-Computer Interaction 0, 0 (2017), 1--37. arXiv:https://doi.org/10.1080/07370024.2017.1344126

[27]

Ting-Hao Kenneth Huang, Walter S. Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. "Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing.

[28]

Ellen A. Isaacs and Herbert H. Clark. 1987. References in conversation between experts and novices. Journal of experimental psychology: general 116, 1 (1987), 26.

[29]

Srinivasan Janarthanam and Oliver Lemon. 2010. Learning to adapt to unknown users: referring expression generation in spoken dialogue systems. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 69--78.

Digital Library

[30]

Laurent Karsenty. 1999. Cooperative work and shared visual context: An empirical study of comprehension problems in side-by-side and remote help dialogues. Human-Computer Interaction 14, 3 (1999), 283--315.

Digital Library

[31]

Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P. Bigham, Jaime Teevan, Ece Kamar, and Walter S. Lasecki.2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation (HCOMP 2017)., HCOMP, Vol. 17.

[32]

Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1301--1318.

Digital Library

[33]

Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing Complex Work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 43--52.

Digital Library

[34]

Robert E. Kraut, Susan R. Fussell, and Jane Siegel. 2003. Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human-Computer Interaction 18, 1--2 (2003), 13--49.

Digital Library

[35]

Walter S. Lasecki and Jeffrey P Bigham. 2013. Interactive crowds: Real-time crowdsourcing and crowd agents. In Handbook of human computation. Springer, 509--521.

[36]

Walter S. Lasecki, Mitchell Gordon, Danai Koutra, Malte F. Jung, Steven P. Dow, and Jeffrey P. Bigham. 2014. Glance: Rapidly Coding Behavioral Video with the Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 551--562.

Digital Library

[37]

Walter S. Lasecki, Mitchell Gordon, Winnie Leung, Ellen Lim, Jeffrey P. Bigham, and Steven P. Dow. 2015. Exploring Privacy and Accuracy Trade-Offs in Crowdsourced Behavioral Video Coding. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1945--1954.

Digital Library

[38]

Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1925--1934.

Digital Library

[39]

Walter S. Lasecki, Kyle I. Murray, Samuel White, Robert C. Miller, and Jeffrey P. Bigham. 2011. Real-time Crowd Control of Existing Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology(UIST '11). ACM, New York, NY, USA, 23--32.

Digital Library

[40]

Walter S. Lasecki, Phyo Thiha, Yu Zhong, Erin Brady, and Jeffrey P. Bigham. 2013. Answering Visual Questions with Conversational Crowd Assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). ACM, New York, NY, USA, Article 18, 8 pages.

Digital Library

[41]

Walter S. Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F. Allen, and Jeffrey P. Bigham. 2013. Chorus:A Crowd-powered Conversational Assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 151--162.

Digital Library

[42]

Matthew Lease, Jessica Hullman, Jeffrey P. Bigham, Michael S. Bernstein, Juho Kim,Walter S. Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert C Miller. 2013. Mechanical turk is not anonymous. (2013).

[43]

Sang Won Lee. 2018. Improving User Involvement through live collaborative creation. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI, USA.

[44]

Sang Won Lee, Yan Chen, and Walter S Lasecki. 2017. The Need for Real-Time Crowd Generation of Task Lists from Speech. In Work-in-Progress, the AAAI Conference on Human Computation (HCOMP 2017).

[45]

Sang Won Lee, Y. Zhang, I. Wong, Yang Y., S. O'Keefe, and W.S. Lasecki. 2017. SketchExpress: Remixing Animations For More Effective Crowd-Powered Prototyping Of Interactive Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). ACM.

Digital Library

[46]

Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2009. TurKit: Tools for Iterative Tasks on Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09). ACM, New York, NY, USA,29--30.

Digital Library

[47]

Alan Lundgard, Yiwei Yang, Maya L. Foster, and Walter S. Lasecki. 2018. Bolt: Instantaneous Crowdsourcing via Just-in-Time Training. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18).ACM, New York, NY, USA, Article 467, 7 pages.

Digital Library

[48]

Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines, and Frederic Vernier. 2004. Beyond "Social Protocols": Multi-user Coordination Policies for Co-located Groupware. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 262--265.

Digital Library

[49]

Michael Nebeling, Alexandra To, Anhong Guo, Adrian A. de Freitas, Jaime Teevan, Steven P. Dow, and Jeffrey P. Bigham.2016. WearWrite: Crowd-Assisted Writing from Smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3834--3846.

Digital Library

[50]

Jakob Nielsen and Thomas K. Landauer. 1993. A Mathematical Model of the Finding of Usability Problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, New York, NY,USA, 206--213.

Digital Library

[51]

Sharon Oviatt, Phil Cohen, LizhongWu, Lisbeth Duncan, Bernhard Suhm, Josh Bers, Thomas Holzman, Terry Winograd, James Landay, Jim Larson, and David Ferro. 2000. Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human-Computer Interaction 15, 4(2000), 263--322.

Digital Library

[52]

Daniel Recktenwald. 2017. Toward a transcription and analysis of live streaming on Twitch. Journal of Pragmatics 115(2017), 68 -- 81.

[53]

Fraser J. M. Reid and Susan E. Reed. 2007. Conversational grounding and visual access in collaborative design. CoDesign3, 2 (2007), 111--122.

[54]

Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 75--85.

Digital Library

[55]

Niloufar Salehi, Andrew McCabe, Melissa Valentine, and Michael Bernstein. 2017. Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1700--1713.

Digital Library

[56]

Niloufar Salehi, Jaime Teevan, Shamsi Iqbal, and Ece Kamar. 2017. Communicating Context to the Crowd for Complex Writing Tasks. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing(CSCW '17). ACM, New York, NY, USA, 1890--1901.

Digital Library

[57]

Stacey D. Scott, M. Sheelagh T. Carpendale, and Kori M. Inkpen. 2004. Territoriality in Collaborative Tabletop Workspaces. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW '04). ACM, New York, NY, USA, 294--303.

Digital Library

[58]

Anthony Tang, Michel Pahud, Kori Inkpen, Hrvoje Benko, John C. Tang, and Bill Buxton. 2010. Three's Company: Understanding Communication Channels in Three-way Distributed Collaboration. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 271--280.

Digital Library

[59]

John C Tang. 1989. Listing, drawing and gesturing in design: A study of the use of shared workspaces by design teams. Ph.D. Thesis (1989).

[60]

John C Tang. 1991. Findings from observational studies of collaborative work. International Journal of Man-machine studies 34, 2 (1991), 143--160.

Digital Library

[61]

Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017.Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3523--3537.

Digital Library

[62]

Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In Proceedings oft he 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York,NY, USA, 1005--1017.

Digital Library

Cited By

Kim GYeo DLee JKim S(2023)Simulating Urban Element Design with Pedestrian Attention: Visual Saliency as Aid for More Visible Wayfinding DesignLand10.3390/land1202039412:2(394)Online publication date: 1-Feb-2023
https://doi.org/10.3390/land12020394
Correia AGrover ASchneider DPimentel AChaves Rde Almeida MFonseca B(2023)Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine InteractionApplied Sciences10.3390/app1304219813:4(2198)Online publication date: 8-Feb-2023
https://doi.org/10.3390/app13042198
Aghayi ELaToza T(2023)A controlled experiment on the impact of microtasking on programmingEmpirical Software Engineering10.1007/s10664-022-10226-228:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10226-2
Show More Cited By

Index Terms

Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool
1. Human-centered computing
2. Information systems
  1. Information systems applications
    1. Collaborative and social computing systems and tools
      1. Synchronous editors

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Web

Within the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 2, Issue CSCW

November 2018

4104 pages

EISSN:2573-0142

DOI:10.1145/3290265

Editors:
Karrie Karahalios
University of Illinois & Adobe
,
Andrés Monroy-Hernández
Snap Inc.
,
Airi Lampinen
Stockholm University
,
Geraldine Fitzpatrick
Vienna University of Technology

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2018

Published in PACMHCI Volume 2, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
650
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)8

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim GYeo DLee JKim S(2023)Simulating Urban Element Design with Pedestrian Attention: Visual Saliency as Aid for More Visible Wayfinding DesignLand10.3390/land1202039412:2(394)Online publication date: 1-Feb-2023
https://doi.org/10.3390/land12020394
Correia AGrover ASchneider DPimentel AChaves Rde Almeida MFonseca B(2023)Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine InteractionApplied Sciences10.3390/app1304219813:4(2198)Online publication date: 8-Feb-2023
https://doi.org/10.3390/app13042198
Aghayi ELaToza T(2023)A controlled experiment on the impact of microtasking on programmingEmpirical Software Engineering10.1007/s10664-022-10226-228:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10226-2
Ma GYang J(2022)Analysis of the Application Prospect of Computer Image Processing Technology in UI DesignAdvances in Multimedia10.1155/2022/59183992022Online publication date: 24-Jun-2022
https://dl.acm.org/doi/10.1155/2022/5918399
Méndez Méndez ACartwright MBello JNov O(2022)Eliciting Confidence for Improving Crowdsourced Audio AnnotationsProceedings of the ACM on Human-Computer Interaction10.1145/35129356:CSCW1(1-25)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512935
Wallace SCai TLe BLeiva L(2022)Debiased Label Aggregation for Subjective Crowdsourcing TasksCHI Conference on Human Factors in Computing Systems Extended Abstracts10.1145/3491101.3519614(1-8)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519614
Cheng PMi JLiu Y(2022)Innovative Application of Game Operation Interface UI Design under the Background of Internet of Things Intelligent Technology2022 International Conference on Knowledge Engineering and Communication Systems (ICKES)10.1109/ICKECS56523.2022.10060817(1-5)Online publication date: 28-Dec-2022
https://doi.org/10.1109/ICKECS56523.2022.10060817
You JPark DSong JSuh B(2022)A Labeling Task Design for Supporting Recent Algorithmic Needs2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020415(2689-2698)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020415
Asthana STobar Thommel SHalfaker ABanovic N(2021)Automatically Labeling Low Quality Content on Wikipedia By Leveraging Patterns in Editing BehaviorsProceedings of the ACM on Human-Computer Interaction10.1145/34795035:CSCW2(1-23)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479503
Choi YMonserrat TPark JShin HLee NKim J(2021)ProtoChatProceedings of the ACM on Human-Computer Interaction10.1145/34329244:CSCW3(1-27)Online publication date: 5-Jan-2021
https://dl.acm.org/doi/10.1145/3432924
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents