research-article

Gaze + pinch interaction in virtual reality

Authors:

Benedikt Mayer,

Diako Mardanbegi,

Hans GellersenAuthors Info & Claims

SUI '17: Proceedings of the 5th Symposium on Spatial User Interaction

Pages 99 - 108

https://doi.org/10.1145/3131277.3132180

Published: 16 October 2017 Publication History

Abstract

Virtual reality affords experimentation with human abilities beyond what's possible in the real world, toward novel senses of interaction. In many interactions, the eyes naturally point at objects of interest while the hands skilfully manipulate in 3D space. We explore a particular combination for virtual reality, the Gaze + Pinch interaction technique. It integrates eye gaze to select targets, and indirect freehand gestures to manipulate them. This keeps the gesture use intuitive like direct physical manipulation, but the gesture's effect can be applied to any object the user looks at --- whether located near or far. In this paper, we describe novel interaction concepts and an experimental system prototype that bring together interaction technique variants, menu interfaces, and applications into one unified virtual experience. Proof-of-concept application examples were developed and informally tested, such as 3D manipulation, scene navigation, and image zooming, illustrating a range of advanced interaction capabilities on targets at any distance, without relying on extra controller devices.

Supplementary Material

MP4 File (p99-pfeuffer.mp4)

Download
22.23 MB

MP4 File (p99-pfeuffer_presentation.mp4)

Download
480.34 MB

References

[1]

Hrvoje Benko and Andrew D. Wilson. 2010. Pinch-the-sky Dome: Freehand Multi-point Interactions with Immersive Omni-directional Data. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York, NY, USA, 3045--3050.

Digital Library

[2]

Doug A. Bowman and Larry F. Hodges. 1997. An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments. In Proc. Symposium on Interactive 3D Graphics (I3D '97). ACM, New York, USA, 35--38.

Digital Library

[3]

Doug A. Bowman, Donald B. Johnson, and Larry F. Hodges. 1999. Testbed Evaluation of Virtual Environment Interaction Techniques. In Proc. Symposium on Virtual Reality Software and Technology (VRST '99). ACM, New York, USA, 26--33.

Digital Library

[4]

Doug A. Bowman and Chadwick A. Wingrave. 2001. Design and Evaluation of Menu Systems for Immersive Virtual Environments. In Proc. Virtual Reality Conference (VR '01). IEEE Computer Society, Washington, DC, USA, 149--156. http://dl.acm.org/citation.cfm?id=580521.835855

[5]

Doug A. Bowman, Chadwick A. Wingrave, Joshua M. Campbell, and Vinh Q. Ly. 2001. Using Pinch Gloves for both Natural and Abstract Interaction Techniques in Virtual Environments. In Proc. HCI International. 629--633.

[6]

William A. S. Buxton. 1995. Human-computer Interaction. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, Chapter Chunking and Phrasing and the Design of Human-computer Dialogues, 494--499. http://dl.acm.org/citation.cfm?id=212925.212970

[7]

Ishan Chatterjee, Robert Xiao, and Chris Harrison. 2015. Gaze+Gesture: Expressive, Precise and Targeted Free-Space Interactions. In Proc. International Conference on Multimodal Interaction. ACM, 131--138.

Digital Library

[8]

Nathan Cournia, John D. Smith, and Andrew T. Duchowski. 2003. Gaze- vs. Hand-based Pointing in Virtual Environments. In Extended Abstracts on Human Factors in Computing Systems (CHI EA '03). ACM, New York, USA, 772--773.

Digital Library

[9]

Shujie Deng, Nan Jiang, Jian Chang, Shihui Guo, and Jian J Zhang. 2017. Understanding the impact of multimodal interaction using gaze informed mid-air gesture control in 3D virtual objects manipulation. International Journal of Human-Computer Studies 105 (2017), 68--80.

[10]

Anna Maria Feit, Shane Williams, Arturo Toledo, Ann Paradiso, Harish Kulkarni, Shaun Kane, and Meredith Ringel Morris. 2017. Toward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design. In Proc. CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1118--1130.

Digital Library

[11]

Yves Guiard. 1987. Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model. In Journal of Motor Behavior, Vol. 19. 486--517.

[12]

Ken Hinckley, Randy Pausch, John C. Goble, and Neal F. Kassell. 1994. A Survey of Design Issues in Spatial Input. In Proc. 7th Annual Symposium on User Interface Software and Technology (UIST '94). ACM, New York, USA, 213--222.

Digital Library

[13]

HTC VIVE. 2017. http://www.vive.com.

[14]

Robert J. K. Jacob. 1990. What You Look at is What You Get: Eye Movement-based Interaction Techniques. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '90). ACM, New York, USA, 11--18.

Digital Library

[15]

Robert J K Jacob. 1993. Eye Movement-Based Human-Computer Interaction Techniques: Toward Non-Command Interfaces. In Advances in Human-Computer Interaction. Vol. 4, Ablex Publishing, 151--190.

[16]

Jacek Jankowski and Martin Hachet. 2013. A Survey of Interaction Techniques for Interactive 3D Environments. In Eurographics 2013 - STAR, M. Sbert and L. Szirmay-Kalos (Eds.). The Eurographics Association.

[17]

Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '94). ACM, New York, USA, 258--264.

Digital Library

[18]

Leap Motion. 2017. https://www.leapmotion.com/product/vr.

[19]

Jiandong Liang and Mark Green. 1994. JDCAD: A highly interactive 3D modeling system. Computers & Graphics 18, 4 (1994), 499--506.

[20]

Paul Lubos, Gerd Bruder, Oscar Ariza, and Frank Steinicke. 2016. Touching the Sphere: Leveraging Joint-Centered Kinespheres for Spatial User Interaction. In Proc. Symposium on Spatial User Interaction (SUI '16). ACM, New York, USA, 13--22.

Digital Library

[21]

Jock D. Mackinlay, Stuart K. Card, and George G. Robertson. 1990. Rapid Controlled Movement Through a Virtual 3D Workspace. In Proc. 17th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '90). ACM, New York, USA, 171--176.

Digital Library

[22]

Microsoft Hololens. 2017. https://www.microsoft.com/microsoft-hololens.

[23]

Mark R. Mine. 1995. Virtual Environment Interaction Techniques. Technical Report. Chapel Hill, NC, USA.

[24]

Mark R. Mine, Frederick P. Brooks, Jr., and Carlo H. Sequin. 1997. Moving Objects in Space: Exploiting Proprioception in Virtual-environment Interaction. In Proc. 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., New York, USA, 19--26.

Digital Library

[25]

Oculus Rift. 2017. https://www.oculus.com/.

[26]

Ken Pfeuffer, Jason Alexander, Ming Ki Chong, and Hans Gellersen. 2014. Gaze-touch: Combining Gaze with Multi-touch for Interaction on the Same Surface. In Proc. 27th Annual Symposium on User Interface Software and Technology (UIST '14). ACM, New York, USA, 509--518.

Digital Library

[27]

Ken Pfeuffer, Jason Alexander, Ming Ki Chong, Yanxia Zhang, and Hans Gellersen. 2015. Gaze-Shifting: Direct-Indirect Input with Pen and Touch Modulated by Gaze. In Proc. 28th Annual Symposium on User Interface Software & Technology (UIST '15). ACM, New York, USA, 373--383.

Digital Library

[28]

Ken Pfeuffer, Jason Alexander, and Hans Gellersen. 2016. Partially-indirect Bimanual Input with Gaze, Pen, and Touch for Pan, Zoom, and Ink Interaction. In Proc. CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, USA, 2845--2856.

Digital Library

[29]

Ken Pfeuffer and Hans Gellersen. 2016. Gaze and Touch Interaction on Tablets. In Proc. 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, USA, 301--311.

Digital Library

[30]

Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J. Conway, Seung Hong, Robert C. Zeleznik, and Mark R. Mine. 1997. Image Plane Interaction Techniques in 3D Immersive Environments. In Proc. Symposium on Interactive 3D Graphics (I3D '97). ACM, New York, USA, 39--43.

Digital Library

[31]

Thammathip Piumsomboon, Gun Lee, Robert Lindeman, and Mark Billinghurst. 2017. Exploring Natural Eye-gaze-based Interaction for Immersive Virtual Reality. In IEEE Symposium on 3D User Interfaces (3DUI). 36--39.

[32]

Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst, and Tadao Ichikawa. 1996. The Go-go Interaction Technique: Non-linear Mapping for Direct Manipulation in VR. In Proc. 9th Annual Symposium on User Interface Software and Technology (UIST '96). ACM, New York, USA, 79--80.

Digital Library

[33]

I. Poupyrev, T. Ichikawa, S. Weghorst, and M. Billinghurst. 1998. Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques. Computer Graphics Forum (1998).

[34]

I. Poupyrev, N. Tomokazu, and S. Weghorst. 1998. Virtual Notepad: handwriting in immersive VR. In IEEE Virtual Reality Annual International Symposium. 126--132.

[35]

Pupil. 2017. https://pupil-labs.com/vr-ar/.

[36]

G. Ren and E. O'Neill. 2012. 3D Marking menu selection with freehand gestures. In 2012 IEEE Symposium on 3D User Interfaces (3DUI). 61--68.

[37]

G. Schmidt, Y. Baillot, D. G. Brown, E. B. Tomlin, and J. E. Swan. 2006. Toward Disambiguating Multiple Selections for Frustum-Based Pointing. In 3D User Interfaces (3DUI '06). 87--94.

Digital Library

[38]

Baris Serim and Giulio Jacucci. 2016. Pointing While Looking Elsewhere: Designing for Varying Degrees of Visual Guidance During Manual Input. In Proc. CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, USA, 5789--5800.

Digital Library

[39]

Linda E. Sibert and Robert J. K. Jacob. 2000. Evaluation of Eye Gaze Interaction. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '00). ACM, New York, USA, 281--288.

Digital Library

[40]

Sophie Stellmach and Raimund Dachselt. 2012. Designing Gaze-based User Interfaces for Steering in Virtual Environments. In Proc. Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, USA, 131--138.

Digital Library

[41]

Sophie Stellmach and Raimund Dachselt. 2012. Investigating Gaze-supported Multimodal Pan and Zoom. In Proc. Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, USA, 357--360.

Digital Library

[42]

Richard Stoakley, Matthew J. Conway, and Randy Pausch. 1995. Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, USA, 265--272.

Digital Library

[43]

Vildan Tanriverdi and Robert J. K. Jacob. 2000. Interacting with Eye Movements in Virtual Environments. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '00). ACM, New York, USA, 265--272.

Digital Library

[44]

Jayson Turner, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2015. Gaze+RST: Integrating Gaze and Multitouch for Remote Rotate-Scale-Translate Tasks. In Proc. 33rd Annual Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, USA, 4179--4188.

Digital Library

[45]

Eduardo Velloso, Jayson Turner, Jason Alexander, Andreas Bulling, and Hans Gellersen. 2015. An empirical investigation of gaze selection in mid-air gestural 3D manipulation. Springer International Publishing, 315--330.

[46]

L. Vlaming, J. Smit, and T. Isenberg. 2008. Presenting using two-handed interaction in open space. In 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems. 29--32.

[47]

Colin Ware and Harutune H. Mikaelian. 1987. An Evaluation of an Eye Tracker As a Device for Computer Input. In Proc. SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (CHI '87). ACM, New York, USA, 183--188.

Digital Library

[48]

Kent Watsen, Rudolph P. Darken, and Michael V. Capps. 1999. A Handheld Computer as an Interaction Device to a Virtual Environment. (1999).

[49]

Andrew D. Wilson. 2009. Simulating Grasping Behavior on an Imaging Interactive Surface. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 125--132.

Digital Library

[50]

Andrew D. Wilson, Shahram Izadi, Otmar Hilliges, Armando Garcia-Mendoza, and David Kirk. 2008. Bringing Physics to the Surface. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 67--76.

Digital Library

[51]

Robert C Zeleznik, Andrew S Forsberg, and Jürgen P Schulze. 2005. Look-that-there: Exploiting gaze in virtual reality interactions. Technical Report.

[52]

Shumin Zhai, Carlos Morimoto, and Steven Ihde. 1999. Manual and Gaze Input Cascaded (MAGIC) Pointing. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, USA, 246--253.

Digital Library

Cited By

Zhang JChen TGong WLiu JChen J(2024)Exploring Data Input Problems in Mixed Reality Environments: Proposal and Evaluation of Natural Interaction TechniquesFuture Internet10.3390/fi1605015016:5(150)Online publication date: 27-Apr-2024
https://doi.org/10.3390/fi16050150
Yashiro YWang GHatori FYabuki N(2024)A Study on Differences in Educational Method to Periodic Inspection Work of Nuclear Power PlantsCivilEng10.3390/civileng50300405:3(760-784)Online publication date: 9-Sep-2024
https://doi.org/10.3390/civileng5030040
Ghasemi YChattopadhyay DJeong HKim HHuang J(2024)Effects of Self-Learning and Exploration for XR-based InteractionsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/10711813241265075Online publication date: 9-Sep-2024
https://doi.org/10.1177/10711813241265075
Show More Cited By

Recommendations

The Eye in Extended Reality: A Survey on Gaze Interaction and Eye Tracking in Head-worn Extended Reality
With innovations in the field of gaze and eye tracking, a new concentration of research in the area of gaze-tracked systems and user interfaces has formed in the field of Extended Reality (XR). Eye trackers are being used to explore novel forms of spatial ...
Pinch, Click, or Dwell: Comparing Different Selection Techniques for Eye-Gaze-Based Pointing in Virtual Reality
ETRA '21 Short Papers: ACM Symposium on Eye Tracking Research and Applications

While a pinch action is gaining popularity for selection of virtual objects in eye-gaze-based systems, it is still unknown how well this method performs compared to other popular alternatives, e.g., a button click or a dwell action. To determine pinch’...
Gaze and Touch Interaction on Tablets
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

We explore how gaze can support touch interaction on tablets. When holding the device, the free thumb is normally limited in reach, but can provide an opportunity for indirect touch input. Here we propose gaze and touch input, where touches redirect to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SUI '17: Proceedings of the 5th Symposium on Spatial User Interaction

October 2017

167 pages

ISBN:9781450354868

DOI:10.1145/3131277

General Chair:
Adalberto L. Simeone
KU Leuven, Belgium
,
Program Chairs:
Kyle Johnsen
University of Georgia
,
Rob Teather
Carleton University, Canada
,
Christian Sandor
Nara Institute of Science and Technology (NAIST), Japan

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SUI '17

Sponsor:

SIGCHI

SUI '17: Symposium on Spatial User Interaction

October 16 - 17, 2017

Brighton, United Kingdom

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

186
Total Citations
View Citations
3,140
Total Downloads

Downloads (Last 12 months)900
Downloads (Last 6 weeks)110

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang JChen TGong WLiu JChen J(2024)Exploring Data Input Problems in Mixed Reality Environments: Proposal and Evaluation of Natural Interaction TechniquesFuture Internet10.3390/fi1605015016:5(150)Online publication date: 27-Apr-2024
https://doi.org/10.3390/fi16050150
Yashiro YWang GHatori FYabuki N(2024)A Study on Differences in Educational Method to Periodic Inspection Work of Nuclear Power PlantsCivilEng10.3390/civileng50300405:3(760-784)Online publication date: 9-Sep-2024
https://doi.org/10.3390/civileng5030040
Ghasemi YChattopadhyay DJeong HKim HHuang J(2024)Effects of Self-Learning and Exploration for XR-based InteractionsProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/10711813241265075Online publication date: 9-Sep-2024
https://doi.org/10.1177/10711813241265075
Faleel SSodhi RIrani P(2024)Comparison of Unencumbered Interaction Technique for Head-Mounted DisplaysProceedings of the ACM on Human-Computer Interaction10.1145/36981468:ISS(500-516)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698146
Wagner UAlbrecht MJacobsen AWang HGellersen HPfeuffer K(2024)Gaze, Wall, and Racket: Combining Gaze and Hand-Controlled Plane for 3D Selection in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36981348:ISS(189-213)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698134
Shi RWei YHu XLiu YYue YYu LLiang H(2024)Experimental Analysis of Freehand Multi-object Selection Techniques in Virtual Reality Head-Mounted DisplaysProceedings of the ACM on Human-Computer Interaction10.1145/36981298:ISS(93-111)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698129
Shapiro KQuijada Leyton PStemple LUberti D(2024)Interacting from Afar: A Study of the Relationship Between Usability and Presence in Object Selection at a DistanceProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697305(1-9)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697305
Wagner UJacobsen AFeuchtner TGellersen HPfeuffer K(2024)Eye-Hand Movement of Objects in Near SpaceCompanion Proceedings of the 2024 Conference on Interactive Surfaces and Spaces10.1145/3696762.3698044(18-21)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3696762.3698044
Mathis FMyers BLafreniere BGlueck MMarques D(2024)MR-Driven Near-Future Realities: Previewing Everyday Life Real-World Experiences Using Mixed RealityProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685748(76-85)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685748
Joos LDurdu UWieland JReiterer HKeim DFuchs JFischer M(2024)Evaluating Node Selection Techniques for Network Visualizations in Virtual RealityProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682102(1-11)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682102
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents