tutorial

Evaluation of challenges in human subject studies "in-the-wild" using subjects' personal smartphones

Authors:

Mattia Gustarini,

Selim Ickin,

Katarzyna WacAuthors Info & Claims

UbiComp '13 Adjunct: Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication

Pages 1447 - 1456

https://doi.org/10.1145/2494091.2496041

Published: 08 September 2013 Publication History

Get Access

Abstract

The experimental setting of Human Mobile Computer Interaction (HCI) studies is moving from the controlled laboratory to the user's daily-life environments, while employing the users' own smartphones. These studies are challenging for both new and expert researchers in human subject studies in the HCI field. Within the last three years, we conducted three different smartphone-based user studies. From these studies, we have derived key challenges that we successfully overcame during their execution. In this paper, we present the outcomes and explain the adopted solutions for the challenges identified in the design, development and execution, and data analysis phases during the user studies. Our goal is to give newcomers and junior researchers a practical view on our conducted studies, and help practitioners to reflect on their own studies and possibly apply the proposed solutions.

References

[1]

Baraković, S. and Skorin-Kapov, L. Survey and Challenges of QoE Management Issues in Wireless Networks. Journal of Computer Networks and Communications 2013, (2013), 28.

Crossref

Google Scholar

[2]

Braunstein, A., Granka, L., and Staddon, J. Indirect content privacy surveys: measuring privacy without asking about it. Symposium on Usable Privacy and Security, (2011).

Digital Library

Google Scholar

[3]

BS.1534--1, I.-R.R. Method for the Subjective Assessment of Intermediate Quality Level of Coding Systems. 2003.

Google Scholar

[4]

Dey, A., Wac, K., and Ferreira, D. Getting Closer: An Empirical Investigation of the Proximity of Users to their Smart Phones. Ubicomp, (2011).

Digital Library

Google Scholar

[5]

Dindia, Kathryn; Allen, M. Sex differences in self-disclosure: A meta-analysis. Psychological Bulletin 112, (1992), 106--124.

Crossref

Google Scholar

[6]

Dix, A., Finlay, J.E., Abowd, G.D., and Beale, R. Human-Computer Interaction. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2003.

Digital Library

Google Scholar

[7]

Earl, O. The challenges in large-scale smartphone user studies. HotPlanet, (2010), 5.

Google Scholar

[8]

Ferreira, D., Kostakos, V., and Dey, A. Lessons Learned from Large-Scale User Studies: Using Android Market as a Source of Data. IJMHCI 4, (2012).

Digital Library

Google Scholar

[9]

Ickin, S., Wac, K., Fiedler, M., Janowski, L., Hong, J.-H., and Dey, A.K. Factors influencing quality of experience of commonly used mobile applications. IEEE Communications Magazine, April (2012), 48--56.

Crossref

Google Scholar

[10]

ITU-T Rec. E.800. Definitions of Terms Related to Quality of Service. 2008.

Google Scholar

[11]

Kahneman, D., Krueger, A.B., Schkade, D. a, Schwarz, N., and Stone, A. a. A survey method for characterizing daily life experience: the day reconstruction method. Science (New York, N.Y.) 306, 5702 (2004), 1776--80.

Crossref

Google Scholar

[12]

Patel, S., Kientz, J., and Hayes, G. Farther than you may think: An empirical investigation of the proximity of users to their mobile phones. UbiComp, (2006), 123--140.

Digital Library

Google Scholar

[13]

Raento, M., Oulasvirta, a., and Eagle, N. Smartphones: An Emerging Tool for Social Scientists. Sociological Methods & Research 37, 3 (2009), 426--454.

Crossref

Google Scholar

[14]

Serral-Gracià, R., Cerqueira, E., Curado, M., Yannuzzi, M., Monteiro, E., and Masip-Bruin, X. An overview of quality of experience measurement challenges for video applications in IP networks. WWIC, (2010), 252--263.

Digital Library

Google Scholar

Cited By

View all

De Masi AWac K(2023)Forecasting Smartphone Application Chains: an App-Rank Based ApproachProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627802(87-98)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627802
Sajjadi PEdwards CZhao JFatemi ALong JKlippel AMasterson T(2022)Remote iVR for Nutrition Education: From Design to EvaluationFrontiers in Computer Science10.3389/fcomp.2022.9271614Online publication date: 29-Jun-2022
https://doi.org/10.3389/fcomp.2022.927161
De Masi AWac K(2022)The Importance of Smartphone Connectivity in Quality of LifeQuantifying Quality of Life10.1007/978-3-030-94212-0_23(523-551)Online publication date: 14-Apr-2022
https://doi.org/10.1007/978-3-030-94212-0_23
Show More Cited By

Index Terms

Evaluation of challenges in human subject studies "in-the-wild" using subjects' personal smartphones
1. Information systems

Recommendations

Controlling In-the-Wild Evaluation Studies of Public Displays
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

In this paper, we investigate the potential of controlled in-the-wild studies as an evaluation methodology that merges the benefits of lab-based and in-the-wild studies. Our exploratory investigation builds upon a comparative, between subject experiment ...
Recruiting Older Adults in the Wild: Reflections on Challenges and Lessons Learned from Research Experience
PervasiveHealth '18: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare

It is important to understand the older adults prior to the design process. The understanding can better facilitate design conversations between the researchers and the older adults. In this paper, we discussed our experiences of building a relationship ...
Technical challenges to deliver sensor-based psychological interventions using smartphones
UbiComp/ISWC '19 Adjunct: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers

Research in the development of tools to successfully deliver psychological interventions through smartphones is growing rapidly. As the research body grows towards more cutting-edge solutions that utilize the smartphone's advanced technical capabilities,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

UbiComp '13 Adjunct: Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication

September 2013

1608 pages

ISBN:9781450322157

DOI:10.1145/2494091

General Chairs:
Friedemann Mattern
ETH Zurich, CH
,
Silvia Santini
TU Darmstadt, DE
,
Program Chairs:
John F. Canny
UC Berkeley, US
,
Marc Langheinrich
Università della Svizzera italiana, CH
,
Jun Rekimoto
University of Tokyo, JP

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Tutorial

Conference

UbiComp '13

Sponsor:

SIGMOBILE
University of Florida
SIGCHI

UbiComp '13: The 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 8 - 12, 2013

Zurich, Switzerland

Acceptance Rates

UbiComp '13 Adjunct Paper Acceptance Rate 254 of 399 submissions, 64%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
195
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

De Masi AWac K(2023)Forecasting Smartphone Application Chains: an App-Rank Based ApproachProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627802(87-98)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627802
Sajjadi PEdwards CZhao JFatemi ALong JKlippel AMasterson T(2022)Remote iVR for Nutrition Education: From Design to EvaluationFrontiers in Computer Science10.3389/fcomp.2022.9271614Online publication date: 29-Jun-2022
https://doi.org/10.3389/fcomp.2022.927161
De Masi AWac K(2022)The Importance of Smartphone Connectivity in Quality of LifeQuantifying Quality of Life10.1007/978-3-030-94212-0_23(523-551)Online publication date: 14-Apr-2022
https://doi.org/10.1007/978-3-030-94212-0_23
Radiah RMäkelä VPrange SRodriguez SPiening RZhou YKöhle KPfeuffer KAbdelrahman YHoppe MSchmidt AAlt F(2021)Remote VR Studies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDsACM Transactions on Computer-Human Interaction10.1145/347261728:6(1-36)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1145/3472617
De Masi AWac K(2018)You're Using This App for What?Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers10.1145/3267305.3267544(612-617)Online publication date: 8-Oct-2018
https://dl.acm.org/doi/10.1145/3267305.3267544
Ferreira DOkoshi TNishiyama YKoehler CPark JLucero ALukowicz PKrüger ABulling ALim YPatel S(2016)3rd international workshop on ubiquitous mobile instrumentationProceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct10.1145/2968219.2978276(608-611)Online publication date: 12-Sep-2016
https://dl.acm.org/doi/10.1145/2968219.2978276
De Masi ACiman MGustarini MWac KLukowicz PKrüger ABulling ALim YPatel S(2016)mQoL smart labProceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct10.1145/2968219.2971593(635-640)Online publication date: 12-Sep-2016
https://dl.acm.org/doi/10.1145/2968219.2971593
Afshari SMovahhedinia N(2016)QoE assessment of interactive applications in computer networksMultimedia Tools and Applications10.1007/s11042-014-2331-575:2(903-918)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1007/s11042-014-2331-5
Müller HGove JWebb JCheang A(2015)Understanding and Comparing Smartphone and Tablet UseProceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction10.1145/2838739.2838748(427-436)Online publication date: 7-Dec-2015
https://dl.acm.org/doi/10.1145/2838739.2838748
Ferreira DKoehler CKarapanos EKostakos VMattern FSantini SCanny JLangheinrich MRekimoto J(2013)Ubiquitous mobile instrumentationProceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication10.1145/2494091.2496043(1409-1412)Online publication date: 8-Sep-2013
https://dl.acm.org/doi/10.1145/2494091.2496043

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Controlling In-the-Wild Evaluation Studies of Public Displays

Recruiting Older Adults in the Wild: Reflections on Challenges and Lessons Learned from Research Experience

Technical challenges to deliver sensor-based psychological interventions using smartphones