research-article

Investigating Cross-Device Interaction between a Handheld Device and a Large Display

Authors:

Dimitrios Raptis,

Jesper Kjeldskov,

Mikael B. Skov,

Bjarke M. LauridsenAuthors Info & Claims

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Pages 6608 - 6619

https://doi.org/10.1145/3025453.3025724

Published: 02 May 2017 Publication History

Abstract

There is a growing interest in HCI research to explore cross-device interaction, giving rise to an interest in different approaches facilitating interaction between handheld devices and large displays. Contributing to this, we have investigated the use of four existing approaches combining touch and mid-air gestures, pinching, swiping, swinging and flicking. We look specifically at their relative efficiency, effectiveness and accuracy in bi-directional interaction between a smartphone and large display in a point-click context. We report findings from two user studies, which show that swiping is both most effective, fastest and most accurate, closely followed by swinging. What these two approaches have in common is the ability to keep the pointer steady on the large display, unaffected by concurrent gestures or body movements used to complete the interaction, suggesting that this is an important factor for designing effective cross-device interaction with large displays.

Supplementary Material

suppl.mov (pn2359-file3.mp4)

Supplemental video

Download
43.23 MB

References

[1]

Michel Beaudouin-Lafon, Stephane Huot, Mathieu Nancel, Wendy Mackay, Emmanuel Pietriga, Romain Primet, Julie Wagner, Olivier Chapuis, Clement Pillias, James Eagan, Tony Gjerlufsen, Clemens Klokmose. 2012. Multisurface interaction in the wild room. Computer 45, no. 4 (2012): 48--56.

Digital Library

[2]

Hrvoje Benko and Andrew D. Wilson. 2010. Multipoint Interactions with Immersive Omnidirectional Visualizations in a Dome. In ACM International Conference on Interactive Tabletops and Surfaces (ITS'10). ACM, NY, NY, USA, 19--28.

Digital Library

[3]

Richard A. Bolt. 1980. "Put-that-there": Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques (SIGGRAPH '80). ACM, NY, NY, USA, 262--270.

Digital Library

[4]

Sebastian Boring, Marko Jurmu, and Andreas Butz. 2009. Scroll, Tilt or Move It: Using Mobile Phones to Continuously Control Pointers on Large Public Displays. In Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7 (OZCHI '09). ACM, NY, NY, USA, 161--168.

Digital Library

[5]

Andrew Bragdon, Rob DeLine, Ken Hinckley, and Meredith Ringel Morris. 2011. Code Space: Touch + Air Gesture Hybrid Interactions for Supporting Developer Meetings. In Proceedings of the International Conference on Interactive Tabletops and Surfaces (ITS '11). ACM, NY, NY, USA, 212--221.

Digital Library

[6]

Raimund Dachselt and Robert Buchholz. 2009. Natural throw and tilt interaction between mobile phones and distant displays. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, NY, NY, USA, 3253--3258.

Digital Library

[7]

L. Gallo and A. Minutolo. 2012. Design and Comparative Evaluation of Smoothed Pointing: A Velocity-oriented Remote Pointing Enhancement Technique. Int. J. Hum.-Comput. Stud. 70, 4 (April 2012), 287--300.

Digital Library

[8]

Tony Gjerlufsen, Clemens Nylandsted Klokmose, James Eagan, Clément Pillias, and Michel BeaudouinLafon. 2011. Shared substance: developing flexible multi-surface applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 3383--3392.

Digital Library

[9]

Luke Hespanhol, Martin Tomitsch, Kazjon Grace, Anthony Collins, and Judy Kay. 2012. Investigating Intuitiveness and Effectiveness of Gestures for Free Spatial Interaction with Large Displays. In Proceedings of the 2012 International Symposium on Pervasive Displays (PerDis '12). ACM, NY, NY, USA.

Digital Library

[10]

Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 1063--1072.

Digital Library

[11]

Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems CHI '15, ACM Press, 1247--1256.

Digital Library

[12]

Steven Houben, Paolo Tell, and Jakob E. Bardram. 2014. ActivitySpace: Managing Device Ecologies in an Activity-Centric Configuration Space. In Proceedings of International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, NY, NY, USA, 119--128.

Digital Library

[13]

David C. Hoaglin and Boris Iglewicz. 1987. FineTuning Some Resistant Rules for Outlier Labeling. J. Amer. Statist. Assoc. 82, 400 (1987), 1147--1149. http://www.jstor.org/stable/2289392

[14]

Kaori Ikematsu and Itiro Siio. 2015. Memory Stones: An Intuitive Information Transfer Technique Between Multi-touch Computers. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). ACM, NY, NY, USA, 3--8.

Digital Library

[15]

Mikkel R. Jakobsen, Yvonne Jansen, Sebastian Boring, and Kasper Hornbæk. 2015. Should I Stay or Should I Go? Selecting Between Touch and mid-air gestures for large-display interaction. In Proceedings of the 15th IFIP TC 13 International Conference on HumanComputer Interaction (INTERACT 2015), 455--473.

Digital Library

[16]

Tero Jokela, Jarno Ojala, Guido Grassel, Petri Piippo, and Thomas Olsson. 2015. A Comparison of Methods to Move Visual Objects Between Personal Mobile Devices in Different Contexts of Use. In Proceedings of the 17th International Conference on HumanComputer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, NY, NY, USA, 172--181.

Digital Library

[17]

Tero Jokela, Jarno Ojala, and Thomas Olsson. 2015. A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 3903--3912.

Digital Library

[18]

Andrés Lucero, Jussi Holopainen, and Tero Jokela. 2012. MobiComics: Collaborative Use of Mobile Phones and Large Displays for Public Expression. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '12). ACM, NY, NY, USA, 383--392.

Digital Library

[19]

Anders Markussen, Sebastian Boring, Mikkel R. Jakobsen, and Kasper Hornbæk. 2016. Off-Limits: Interacting Beyond the Boundaries of Large Displays. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 5862--5873.

Digital Library

[20]

Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: A Mid-air Wordgesture Keyboard. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 1073--1082.

Digital Library

[21]

Nicolai Marquardt, Ken Hinckley, and Saul Greenberg. 2012. Cross-device interaction via micro-mobility and f-formations. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, NY, NY, USA, 13--22.

Digital Library

[22]

Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12). ACM, NY, NY, USA, 31--40.

Digital Library

[23]

Sven Mayer, Katrin Wolf, Stefan Schneegass, and Niels Henze. 2015. Modeling Distant Pointing for Compensating Systematic Displacements. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 4165--4168.

Digital Library

[24]

Meredith R. Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, m. c. schraefel, and Jacob O. Wobbrock. 2014. Reducing legacy bias in gesture elicitation studies. interactions 21, 3 (May 2014), 40--45.

Digital Library

[25]

Brad A. Myers, Rishi Bhatnagar, Jeffrey Nichols, Choon Hong Peck, Dave Kong, Robert Miller, and A. Chris Long. 2002. Interacting at a Distance: Measuring the Performance of Laser Pointers and Other Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, NY, NY, USA, 33--40.

Digital Library

[26]

Miguel A. Nacenta, Carl Gutwin, Dzmitry Aliakseyeu, Sriram Subramanian. 2009. There and Back again: Cross-Display Object Movement in Multi-Display Environments. Journal of Human-Computer Interaction, vol. 24 no. 1, 170--229.

[27]

Mathieu Nancel, Olivier Chapuis, Emmanuel Pietriga, Xing-Dong Yang, Pourang P. Irani, and Michel Beaudouin-Lafon. 2013. High-precision Pointing on Large Wall Displays Using Small Handheld Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 831--840.

Digital Library

[28]

Roman Rädle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers. 2015. Spatially-aware or Spatially-agnostic? Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 3913--3922.

Digital Library

[29]

Roman Rädle, Hans-Christian Jetter, Nicolai Marquardt, Harald Reiterer, and Yvonne Rogers. 2014. HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, NY, NY, USA, 45--54.

Digital Library

[30]

Dimitrios Raptis, Jesper Kjeldskov, and Mikael B. Skov. 2016. Continuity in Multi-Device Interaction: An Online Study. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction: GameChanging Design (NordiCHI '16). ACM, NY, NY, USA.

Digital Library

[31]

Umar Rashid, Jarmo Kauko, Jonna Häkkilä, and Aaron Quigley. 2011. Proximal and Distal Selection of Widgets: Designing Distributed UI for Mobile Interaction with Large Display. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, NY, NY, USA, 495--498.

Digital Library

[32]

Jun Rekimoto. 1997. Pick-and-drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST '97). ACM, NY, NY, USA, 31--39.

Digital Library

[33]

Jürgen Scheible, Timo Ojala, and Paul Coulton. 2008. MobiToss: A Novel Gesture Based Interface for Creating and Sharing Mobile Multimedia Art on Large Public Displays. In Proceedings of the 16th ACM International Conference on Multimedia (MM '08). ACM, NY, NY, USA, 957--960.

Digital Library

[34]

Julian Seifert, Andreas Bayer, and Enrico Rukzio. 2013. PointerPhone: Using Mobile Phones for Direct Pointing Interactions with Remote Displays. In Proceedings of the 14th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT 2013). Springer Berlin Heidelberg, 18--35.

[35]

Mikael B. Skov, Jesper Kjeldskov, Jeni Paay, Heidi P. Jensen, and Marius P. Olsen. 2015. Investigating Cross-Device Interaction Techniques. Proceedings of the Australian Conference on Human-Computer Interaction, OzCHI '15, ACM Press, 446--454.

Digital Library

[36]

Henrik Sørensen, Dimitrios Raptis, Jesper Kjeldskov, and Mikael B. Skov. 2014. The 4C framework: principles of interaction in digital ecosystems. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, NY, NY, USA, 87--97.

Digital Library

[37]

Daniel Vogel and Ravin Balakrishnan. 2005. Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (UIST '05). ACM, NY, NY, USA, 33--42.

Digital Library

[38]

Robert Walter, Gilles Bailly, Nina Valkanova, and Jörg Müller. 2014. Cuenesics: using mid-air gestures to select items on interactive public displays. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, NY, NY, USA, 299--308.

Digital Library

[39]

Paweł Wozniak, Lars Lischke, Benjamin Schmidt, Shengdong Zhao, and Morten Fjeld. 2014. Thaddeus: A Dual Device Interaction Space for Exploring Information Visualisation. Proceedings of the 8th Nordic Conference on Human-Computer Interaction NordiCHI '14, ACM Press, 41--50.

Digital Library

[40]

Koji Yatani, Koiti Tamura, Keiichi Hiroki, Masanori Sugimoto, and Hiromichi Hashizume. 2005. Toss-it: Intuitive Information Transfer Techniques for Mobile Devices. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, NY, NY, USA, 1881--1884.

Digital Library

Cited By

Bateman SGutwin CMansoor HNacenta Mvan der Kamp MBaliesnyi MGagnon KRollheiser J(2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593236
Li TWu SJin YShi HLiu S(2023)X-Space: Interaction design of extending mixed reality space from Web2D visualizationVisual Informatics10.1016/j.visinf.2023.10.0017:4(73-83)Online publication date: Dec-2023
https://doi.org/10.1016/j.visinf.2023.10.001
Papadoulis GSintoris CFidas CAvouris N(2023)Extending User Interaction with Mixed Reality Through a Smartphone-Based ControllerHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42280-5_27(426-435)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42280-5_27
Show More Cited By

Index Terms

Investigating Cross-Device Interaction between a Handheld Device and a Large Display
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

A Comparison of Techniques for Cross-Device Interaction from Mobile Devices to Large Displays
MoMM '16: Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media

In recent years there has been an increasing interest in cross-device interaction research involving mobile computing. We contribute to this research with a comparative study of four interaction techniques for moving information from a mobile device to ...
A comparison of techniques for cross-device interaction from mobile devices to large displays

In recent years there has been an increasing interest in cross-device interaction research involving mobile computing. We contribute to this research with a comparative study of four interaction techniques for moving information from a mobile device to ...
Eliciting Mid-Air Gestures for Wall-Display Interaction
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

Freehand mid-air gestures are a promising input method for interacting with wall displays. However, work on mid-air gestures for wall-display interaction has mainly explored what is technically possible, which might not result in gestures that users ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

May 2017

7138 pages

ISBN:9781450346559

DOI:10.1145/3025453

General Chairs:
Gloria Mark
University of California Irvine
,
Susan Fussell
Cornell University
,
Program Chairs:
Cliff Lampe
University of Michigan
,
m.c. schraefel
University of Southampton
,
Juan Pablo Hourcade
University of Iowa
,
Caroline Appert
Université Paris-Sud
,
Daniel Wigdor
University of Toronto

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Innovation Fund Denmark

Conference

CHI '17

Sponsor:

SIGCHI

CHI '17: CHI Conference on Human Factors in Computing Systems

May 6 - 11, 2017

Colorado, Denver, USA

Acceptance Rates

CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
1,221
Total Downloads

Downloads (Last 12 months)80
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bateman SGutwin CMansoor HNacenta Mvan der Kamp MBaliesnyi MGagnon KRollheiser J(2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593236
Li TWu SJin YShi HLiu S(2023)X-Space: Interaction design of extending mixed reality space from Web2D visualizationVisual Informatics10.1016/j.visinf.2023.10.0017:4(73-83)Online publication date: Dec-2023
https://doi.org/10.1016/j.visinf.2023.10.001
Papadoulis GSintoris CFidas CAvouris N(2023)Extending User Interaction with Mixed Reality Through a Smartphone-Based ControllerHuman-Computer Interaction – INTERACT 202310.1007/978-3-031-42280-5_27(426-435)Online publication date: 25-Aug-2023
https://doi.org/10.1007/978-3-031-42280-5_27
Babic TReiterer HHaller M(2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
https://doi.org/10.3390/mti6100094
Homaeian LWallace JScott S(2022)Handoff and Deposit: Designing Temporal Coordination in Cross-Device Transfer Techniques for Mixed-Focus CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/35551926:CSCW2(1-23)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555192
Li TWu SJin YShi HLiu S(2022)X-Space: A Tool for Extending Mixed Reality Space from Web2D Visualization Anywhere2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct57072.2022.00032(124-130)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR-Adjunct57072.2022.00032
Peng CDong YCao L(2022)Real-Time Bimanual Interaction Across Virtual WorkspacesVirtual, Augmented and Mixed Reality: Design and Development10.1007/978-3-031-05939-1_23(338-356)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-05939-1_23
Yang JKong JZhao C(2021)A Smartphone-Based Cursor Position System in Cross-Device Interaction Using Machine Learning TechniquesSensors10.3390/s2105166521:5(1665)Online publication date: 28-Feb-2021
https://doi.org/10.3390/s21051665
Saidi HDubois ESerrano MKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)HoloBar: Rapid Command Execution for Head-Worn AR Exploiting Around the Field-of-View InteractionProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445255(1-17)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445255
Suarez GChoquevilca KPadilla AGalvez M(2021)Leaper: Video Game Development on Android Devices with Cross-Device Interaction2021 IEEE Sciences and Humanities International Research Conference (SHIRCON)10.1109/SHIRCON53068.2021.9652333(1-4)Online publication date: 17-Nov-2021
https://doi.org/10.1109/SHIRCON53068.2021.9652333
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents