research-article

Crowdsourcing vs Laboratory-Style Social Acceptability Studies?: Examining the Social Acceptability of Spatial User Interactions for Head-Worn Displays

Authors:

Yumiko Sakamoto,

Khalad HasanAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 310, Pages 1 - 7

https://doi.org/10.1145/3173574.3173884

Published: 21 April 2018 Publication History

Abstract

The use of crowdsourcing platforms for data collection in HCI research is attractive in their ability to provide rapid access to large and diverse participant samples. As a result, several researchers have conducted studies investigating the similarities and differences between data collected through crowdsourcing and more traditional, laboratory-style data collection. We add to this body of research by examining the feasibility of conducting social acceptability studies via crowdsourcing. Social acceptability can be a key determinant for the early adoption of emerging technologies, and as such, we focus our investigation on social acceptability for Head-Worn Display (HWD) input modalities. Our results indicate that data collected via a crowdsourced experiment and a laboratory-style setting did not differ at a statistically significant level. These results provide initial support for crowdsourcing platforms as viable options for conducting social acceptability research.

References

[1]

David Ahlström, Khalad Hasan, and Pourang Irani. 2014. Are You Comfortable Doing That?: Acceptance Studies of Around-Device Gestures in and for Public Settings. Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services - MobileHCI '14: 193--202.

Digital Library

[2]

Merrill Anderson. 2004. Bottom-line organization development. Routledge.

[3]

Florian Bemmann. 2015. User Preference for Smart Glass Interaction. 3--5.

[4]

Sabine Buchholz and Javier Latorre. 2011. Crowdsourcing preference tests, and how to detect cheating. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 3053--3056.

[5]

Brent C. 2011. How Much Time are Respondents Willing to Spend on Your Survey? Retrieved from https://www.surveymonkey.com/blog/2011/02/14/surv ey_completion_times/

[6]

Andrea Colaço, Ahmed Kirmani, Hye Soo Yang, NanWei Gong, Chris Schmandt, and Vivek K Goyal. 2013. Mime: Compact, Low Power 3D Gesture Sensing for Interaction with Head Mounted Displays. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13), 227--236.

Digital Library

[7]

Kristen Dergousoff and Regan L Mandryk. 2015. Mobile Gamification for Crowdsourcing Data Collection: Leveraging the Freemium Model. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1065--1074.

Digital Library

[8]

Mira Dontcheva, Robert R. Morris, Joel R. Brandt, and Elizabeth M. Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14: 3379-- 3388.

Digital Library

[9]

Jennifer Edgar, Joe Murphy, and Michael Keating. 2016. Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions. SAGE Open 6, 4: 2158244016671770.

[10]

Barrett Ens, Ahmad Byagowi, Teng Han, Juan David Hincapié-Ramos, and Pourang Irani. 2016. Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces. In Proceedings of the 2016 Symposium on Spatial User Interaction - SUI '16 (SUI '16), 99--102.

Digital Library

[11]

Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2014. Towards usable and acceptable above-device interactions. In MobileHCI 2014 - Proceedings of the 16th ACM International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '14), 459--464.

Digital Library

[12]

Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1631--1640.

Digital Library

[13]

Hervé Goëau, Alexis Joly, Souheil Selmi, Pierre Bonnet, Elise Mouysset, Laurent Joyeux, Jean-François Molino, Philippe Birnbaum, Daniel Bathelemy, and Nozha Boujemaa. 2011. Visual-based Plant Species Identification from Crowdsourced Data. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11), 813--814.

Digital Library

[14]

Khalad Hasan, David Ahlström, Junhyeok Kim, and Pourang Irani. 2017. AirPanes: Two-Handed AroundDevice Interaction for Pane Switching on Smartphones. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 679--691.

Digital Library

[15]

Khalad Hasan, David Ahlström, and Pourang Irani. 2013. Ad-binning: leveraging around device space for storing, browsing and retrieving mobile device content. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 899--908.

Digital Library

[16]

Caroline Jay and Roger Hubbold. 2003. Amplifying head movements with head-mounted displays. Presence: Teleoperators and Virtual Environments 12, 3: 268--276.

Digital Library

[17]

Brett Jones, Rajinder Sodhi, David Forsyth, Brian Bailey, and Giuliano Maciocci. 2012. Around Device Interaction for Multiscale Navigation. In Proceedings of the 14th International Conference on Humancomputer Interaction with Mobile Devices and Services (MobileHCI '12), 83--92.

Digital Library

[18]

Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. Proceedings of the 2008 SIGCHI Conference on Human Factors in Computing Systems: 453--456.

Digital Library

[19]

Aniket Kittur and Robert E Kraut. 2008. Harnessing the Wisdom of Crowds in Wikipedia: Quality Through Coordination. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08), 37--46.

Digital Library

[20]

Barry Kollee, Sven Kratz, and Anthony Dunnigan. 2014. Exploring Gestural Interaction in Smart Spaces Using Head Mounted Devices with Ego-centric Sensing. In Proceedings of the 2Nd ACM Symposium on Spatial User Interaction (SUI '14), 40--49.

Digital Library

[21]

Steven Komarov, Katharina Reinecke, and Krzysztof Z Gajos. 2013. Crowdsourcing Performance Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 207--216.

Digital Library

[22]

Sven Kratz, Michael Rohs, Dennis Guse, Jörg Müller, Gilles Bailly, and Michael Nischt. 2012. PalmSpace: Continuous Around-device Gestures vs. Multitouch for 3D Rotation Tasks on Mobile Devices. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI '12), 181--188.

Digital Library

[23]

Meethu Malu and Leah Findlater. 2015. Personalized, Wearable Control of a Head-mounted Display for Users with Upper Body Motor Impairments. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 221-- 230.

Digital Library

[24]

Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1: 1--23.

[25]

Daniel McDuff, Rana el Kaliouby, and Rosalind Picard. 2011. Crowdsourced Data Collection of Facial Responses. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI '11), 11-- 18.

Digital Library

[26]

Florian Müller, Niloofar Dezfuli, Max Mühlhäuser, Martin Schmitz, and Mohammadreza Khalilbeigi. 2015. Palm-based Interaction with Head-mounted Displays. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '15), 963--965.

Digital Library

[27]

Julie Rico and Stephen Brewster. 2010. Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 887--896.

Digital Library

[28]

Julie Rico and Stephen Brewster. 2010. Gesture and Voice Prototyping for Early Evaluations of Social Acceptability in Multimodal Interfaces. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI '10), 16:1--16:9.

Digital Library

[29]

Sami Ronkainen, Jonna Häkkilä, Saana Kaleva, Ashley Colley, and Jukka Linjama. 2007. Tap Input As an Embedded Interaction Method for Mobile Devices. In Proceedings of the 1st International Conference on Tangible and Embedded Interaction (TEI '07), 263-- 270.

Digital Library

[30]

Spencer Rothwell, Steele Carter, Ahmad Elshenawy, and Daniela Braga. 2015. Job Complexity and User Attention in Crowdsourcing Microtasks. HCOMP 2015 Workshop Papers for Crowdsourcing Breakthroughs for Language Technology Applications," 2008: 20--25.

[31]

Jaime Ruiz and Yang Li. 2011. DoubleFlip: a motion gesture delimiter for mobile interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2717--2720.

Digital Library

[32]

Marcos Serrano, Barrett M Ens, and Pourang P Irani. 2014. Exploring the Use of Hand-to-face Input for Interacting with Head-worn Displays. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 3181--3190.

Digital Library

[33]

Mark D Smucker and Chandra Prakash Jethani. 2011. The crowd vs. the lab: A comparison of crowd-sourced and university laboratory participant behavior. In Proceedings of the SIGIR 2011 Workshop on crowdsourcing for information retrieval.

[34]

Shichang Wang, Chu-Ren Huang, Yao Yao, and Angel Chan. 2015. Mechanical Turk-based Experiment vs Laboratory-based Experiment: A Case Study on the Comparison of Semantic Transparency Rating Data. In PACLIC.

[35]

Julie R Wiliamson, Andrew Crossan, and Stephen Brewster. 2011. Multimoda Mobile Interactions: Usability Studies in Real World Settings. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI '11), 361--368.

Digital Library

[36]

Julie R Williamson. 2012. User experience, performance, and social acceptability: usable multimodal mobile interaction. University of Glasgow.

[37]

Julie R Williamson, Stephen Brewster, and Rama Vennelakanti. 2013. Mo!Games: Evaluating Mobile Gestures in the Wild. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI '13), 173--180.

Digital Library

Cited By

Zhang QLan YGuo KWang D(2024)Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596148:2(1-29)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659614
Mo WDechant MMarquardt NAyobi ASingh AHolloway C(2024)Exploring the Design Space of Input Modalities for Working in Mixed Reality on Long-haul FlightsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661560(2267-2285)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661560
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Show More Cited By

Index Terms

Crowdsourcing vs Laboratory-Style Social Acceptability Studies?: Examining the Social Acceptability of Spatial User Interactions for Head-Worn Displays
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Performer vs. observer: whose comfort level should we consider when examining the social acceptability of input modalities for head-worn display?
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

The popularity of head-worn displays (HWD) technologies such as Virtual Reality (VR) and Augmented Reality (AR) headsets is growing rapidly. To predict their commercial success, it is essential to understand the acceptability of these new technologies, ...
Exploring online social behavior in crowdsourcing communities

Both I-mode and we-mode collective intentions significantly predicted online social behavior in crowdsourcing communities.We-mode collective intention exerted a greater effect on online social behavior, as compared to I-mode collective ...
Crowdsourcing Social Media for Military Operations
SocialSens'17: Proceedings of the 2nd International Workshop on Social Sensing

In this paper, we consider the demographics associated with social media users as a basis for determining how to interact with a population group to inform military operations such as humanitarian aid and disaster relief (HADR). With social media use ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSERC

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
426
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)1

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang QLan YGuo KWang D(2024)Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596148:2(1-29)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659614
Mo WDechant MMarquardt NAyobi ASingh AHolloway C(2024)Exploring the Design Space of Input Modalities for Working in Mixed Reality on Long-haul FlightsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661560(2267-2285)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661560
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Medeiros DDubus RWilliamson JWilson GPöhlmann KMcGill M(2023)Surveying the Social Comfort of Body, Device, and Environment-Based Augmented Reality Interactions in Confined Passenger Spaces Using Mixed Reality Composite VideosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109237:3(1-25)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610923
Jung IAhn SSeo JHong J(2023)Exploring the Potentials of Crowdsourcing for Gesture Data CollectionInternational Journal of Human–Computer Interaction10.1080/10447318.2023.218023540:12(3112-3121)Online publication date: 23-Feb-2023
https://doi.org/10.1080/10447318.2023.2180235
Rixen JColley MAskari AGugenheimer JRukzio E(2022)Consent in the Age of AR: Investigating The Comfort With Displaying Personal Information in Augmented RealityProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502140(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502140
Hossain TIslam MDelamare WChowdhury FHasan K(2021)Exploring Social Acceptability and Users’ Preferences of Head- and Eye-Based Interaction with Mobile DevicesProceedings of the 20th International Conference on Mobile and Ubiquitous Multimedia10.1145/3490632.3490636(12-23)Online publication date: 5-Dec-2021
https://dl.acm.org/doi/10.1145/3490632.3490636
Logas JLin GBelan KGogate AStarner T(2021)Conversational Partner’s Perception of Subtle Display Use for Monitoring NotificationsProceedings of the Augmented Humans International Conference 202110.1145/3458709.3458942(101-110)Online publication date: 22-Feb-2021
https://dl.acm.org/doi/10.1145/3458709.3458942
Antoine AMalacria SMarquardt NCasiez GKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Interaction Illustration Taxonomy: Classification of Styles and Techniques for Visually Representing Interaction ScenariosProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445586(1-22)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445586
Pandey LHasan KArif AKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Acceptability of Speech and Silent Speech Input Methods in Private and PublicProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445430(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445430
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents