Investigating the Effects of Avatarization and Interaction Techniques on Near-field Mixed Reality Interactions with Physical Components
Authors: 
Roshan Venkatakrishnan, Rohith Venkatakrishnan, Ryan Canales, Balagopal Raveendranath, Christopher C. Pagano, Andrew C. Robb, Wen-Chieh Lin, Sabarish V. BabuAuthors Info & Claims
Pages 2756 - 2766
Abstract
Mixed reality (MR) interactions feature users interacting with a combination of virtual and physical components. Inspired by research investigating aspects associated with near-field interactions in augmented and virtual reality (AR & VR), we investigated how avatarization, the physicality of the interacting components, and the interaction technique used to manipulate a virtual object affected performance and perceptions of user experience in a mixed reality fundamentals of laparoscopic peg-transfer task wherein users had to transfer a virtual ring from one peg to another for a number of trials. We employed a 3 (Physicality of pegs) X 3 (Augmented Avatar Representation) X 2 (Interaction Technique) multi-factorial design, manipulating the physicality of the pegs as a between-subjects factor, the type of augmented self-avatar representation, and the type of interaction technique used for object-manipulation as within-subjects factors. Results indicated that users were significantly more accurate when the pegs were virtual rather than physical because of the increased salience of the task-relevant visual information. From an avatar perspective, providing users with a reach envelope-extending representation, though useful, was found to worsen performance, while co-located avatarization significantly improved performance. Choosing an interaction technique to manipulate objects depends on whether accuracy or efficiency is a priority. Finally, the relationship between the avatar representation and interaction technique dictates just how usable mixed reality interactions are deemed to be.
References
[1]
F. Argelaguet and C. Andujar. A survey of 3d object selection techniques for virtual environments. Computers & Graphics, 37(3):121–136, 2013. 2.
[2]
R. T. Azuma. A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4):355–385, 1997. 1, 2.
[3]
S. K. Badam, A. Srinivasan, N. Elmqvist, and J. Stasko. Affordances of input modalities for visual data exploration in immersive environments. In 2nd Workshop on Immersive Analytics, 2017. 3.
[4]
D. A. Bowman. Interaction techniques for common tasks in immersive virtual environments: design, evaluation, and application. Georgia Institute of Technology, 1999. 5.
[5]
E. Bozgeyikli and L. L. Bozgeyikli. Evaluating object manipulation interaction techniques in mixed reality: Tangible user interfaces and gesture. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 778–787. IEEE, 2021. 3.
[6]
D. Brickler and S. V. Babu. An evaluation of screen parallax, haptic feedback, and sensory-motor mismatch on near-field perception-action coordination in vr. ACM Transactions on Applied Perception (TAP), 18(4):1–16, 2021. 2,4,5,6.
[7]
D. Brickler, R. J. Teather, A. T. Duchowski, and S. V. Babu. A fitts' law evaluation of visuo-haptic fidelity and sensory mismatch on user performance in a near-field disc transfer task in virtual reality. ACM Transactions on Applied Perception (TAP), 17(4):1–20, 2020.5.
[8]
D. Brickler, M. Volonte, J. W. Bertrand, A. T. Duchowski, and S. V. Babu. Effects of stereoscopic viewing and haptic feedback, sensory-motor congruence and calibration on near-field fine motor perception-action coordination in virtual reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 28–37. IEEE, 2019. 2,3,4,5,6,8,9.
[9]
I. M. Bullock, T. Feix, and A. M. Dollar. Human precision manipulation workspace: Effects of object size and number of fingers used. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5768–5772. IEEE, 2015. 8.
[10]
R. Canales and S. Jorg. Performance is not everything: Audio feedback preferred over visual feedback for grasping task in virtual reality. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, pp. 1–6, 2020. 5.
[11]
Z. Chen, J. Li, Y. Hua, R. Shen, and A. Basu. Multimodal interaction in augmented reality. In 2017 IEEE international conference on systems, man, and cybernetics (SMC), pp. 206–209. IEEE, 2017. 2.
[12]
K.-Y. Cheng, R.-H. Liang, B.-Y. Chen, R.-H. Laing, and S.-Y. Kuo. icon: utilizing everyday objects as additional, auxiliary and instant tabletop controllers. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 1155–1164, 2010. 2.
[13]
E. Ebrahimi, A. Robb, L. S. Hartman, C. C. Pagano, and S. V. Babu. Effects of anthropomorphic fidelity of self-avatars on reach boundary estimation in immersive virtual environments. In Proceedings of the 15th ACM Symposium on Applied Perception, pp. 1–8, 2018. 6.
[14]
J. Ehnes. A tangible interface for the ami content linking device-the automated meeting assistant. In 2009 2nd Conference on Human System Interactions, pp. 306–313. IEEE, 2009. 2.
[15]
S. Esmaeili, B. Benda, and E. D. Ragan. Detection of scaled hand interactions in virtual reality: The effects of motion direction and task complexity. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 453–462. IEEE, 2020. 2.
[16]
B. R. Fajen. Visual control of locomotion. Cambridge University Press, 2021. 8.
[17]
A. O. S. Feiner. The flexible pointer: An interaction technique for selection in augmented and virtual reality. In Proc. UIST, vol. 3, pp. 81–82, 2003. 2.
[18]
T. Feuchtner and J. Muller. Extending the body for interaction with reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 5145–5157, 2017. 3.
[19]
G. M. Fried. Fls assessment of competency using simulated laparoscopic tasks. Journal of Gastrointestinal Surgery, 12:210–212, 2008. 3,5.
[20]
M. Frutos-Pascual, C. Creed, and I. Williams. Head mounted display interaction evaluation: manipulating virtual objects in augmented reality. In IFIP Conference on Human-Computer Interaction, pp. 287–308. Springer, 2019. 2.
[21]
A. C. S. Genay, A. Lecuyer, and M. Hachet. Being an avatar” for real”: a survey on virtual embodiment in augmented reality. IEEE Transactions on Visualization and Computer Graphics, 2021. 3.
[22]
L. Gerini, F. Solari, and M. Chessa. A cup of coffee in mixed reality: analysis of movements' smoothness from real to virtual. In 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 566–569. IEEE, 2022. 3.
[23]
S. R. Gomez, R. Jianu, and D. H. Laidlaw. A fiducial-based tangible user interface for white matter tractography. In Advances in Visual Computing: 6th International Symposium, ISVC 2010, Las Vegas, NV, USA, November 29-December 1, 2010, Proceedings, Part II 6, pp. 373–381. Springer, 2010. 3.
[24]
T. Ha, S. Feiner, and W. Woo. Wearhand: Head-worn, rgb-d camera-based, bare-hand user interface with visually enhanced depth perception. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 219–228. IEEE, 2014. 2.
[25]
T. Ha and W. Woo. An empirical evaluation of virtual hand techniques for 3d object manipulation in a tangible augmented reality environment. In 2010 IEEE Symposium on 3D User Interfaces (3DUI), pp. 91–98. IEEE, 2010. 3.
[26]
C. Heinrich, M. Cook, T. Langlotz, and H. Regenbrecht. My hands? importance of personalised virtual hands in a neurorehabilitation scenario. Virtual Reality, 25(2):313–330, 2021. 3.
[27]
A. Hettiarachchi and D. Wigdor. Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1957–1967, 2016. 3.
[28]
T. N. Hoang, R. T. Smith, and B. H. Thomas. Ultrasonic glove input device for distance-based interactions. In 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT), pp. 46–53. IEEE, 2013. 2.
[29]
D. A. Hofmann. An overview of the logic and rationale of hierarchical linear models. Journal of management, 23(6):723–744, 1997. 6.
[30]
H. Ishiyama and S. Kurabayashi. Monochrome glove: A robust real-time hand gesture recognition method by using a fabric glove with design of structured markers. In 2016 IEEE virtual reality (VR), pp. 187–188. IEEE, 2016. 2.
[31]
P. Issartel, F. Gueniat, and M. Ammi. Slicing techniques for handheld augmented reality. In 2014 IEEE symposium on 3D user interfaces (3DUI), pp. 39–42. IEEE, 2014. 3.
[32]
A. S. Johnson and Y. Sun. Spatial augmented reality on person: Exploring the most personal medium. In International Conference on Virtual, Augmented and Mixed Reality, pp. 169–174. Springer, 2013. 3.
[33]
E. Kaiser, A. Olwal, D. McGee, H. Benko, A. Corradini, X. Li, P. Cohen, and S. Feiner. Mutual disambiguation of 3d multimodal interaction in augmented and virtual reality. In Proceedings of the 5th international conference on Multimodal interfaces, pp. 12–19, 2003. 2.
[34]
H. J. Kang, J.-h. Shin, and K. Ponto. A comparative analysis of 3d user interaction: How to move virtual objects in mixed reality. In 2020 IEEE conference on virtual reality and 3D user interfaces (VR), pp. 275–284. IEEE, 2020. 1, 2.
[35]
K. Kohm, J. Porter, and A. Robb. Sensitivity to hand offsets and related behavior in virtual environments over time. ACM Transactions on Applied Perception, 19(4):1–15, 2022. 4.
[36]
R. Kopper, D. A. Bowman, M. G. Silva, and R. P. McMahan. A human motor behavior model for distal pointing tasks. International journal of human-computer studies, 68(10):603–615, 2010. 2.
[37]
P. Kyriakou and S. Hermon. Can i touch this? using natural interaction in a museum augmented reality system. Digital Applications in Archaeology and Cultural Heritage, 12:e00088, 2019. 2.
[38]
M. Kyto, B. Ens, T. Piumsomboon, G. A. Lee, and M. Billinghurst. Pin-pointing: Precise head-and eye-based target selection for augmented reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–14, 2018. 2.
[39]
E. Lamounier K. Lopes, A. Cardoso, and A. Soares. Using augmented reality techniques to simulate myoelectric upper limb prostheses. Journal of Bioengineering & Biomedical Science, 1:010,2012. 3.
[40]
J. R. Lewis. Psychometric evaluation of the pssuq using data from five years of usability studies. International Journal of Human-Computer Interaction, 14(3–4):463–488, 2002. 6.
[41]
K. Li, R. Nataraj, T. L. Marquardt, and Z.-M. Li. Directional coordination of thumb and finger forces during precision pinch. PloS one, 8(11):e79400, 2013. 8.
[42]
Z.-M. Li and J. Tang. Coordination of thumb joints during opposition. Journal of biomechanics, 40(3):502–510, 2007. 8.
[43]
G. Lu, L.-K. Shark, G. Hall, and U. Zeshan. Immersive manipulation of virtual objects through glove-based hand gesture interaction. Virtual Reality, 16(3):243–252, 2012. 2.
[44]
A. Macaranas, A. N. Antle, and B. E. Riecke. What is intuitive interaction? balancing users' performance and satisfaction with natural user interfaces. Interacting with Computers, 27(3):357–370, 2015. 2.
[45]
M. W. Marzke. Precision grips, hand morphology, and tools. American Journal of Physical Anthropology: The Official Publication of the American Association of Physical Anthropologists, 102(1):91–110, 1997. 8.
[46]
D. Merrill and P. Maes. Augmenting looking, pointing and reaching gestures to enhance the searching and browsing of physical objects. In International Conference on Pervasive Computing, pp. 1–18. Springer, 2007. 2.
[47]
D. M. Mifsud, A. S. Williams, F. Ortega, and R. J. Teather. Augmented reality fitts' law input comparison between touchpad, pointing gesture, and raycast. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 590–591. IEEE, 2022. 3.
[48]
P. Milgram, H. Takemura, A. Utsumi, and F. Kishino. Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence technologies, vol. 2351, pp. 282–292. Spie, 1995. 1.
[49]
T. Nagel and F. Heidmann. Exploring faceted geo-spatial data with tangible interaction. GeoViz 2011, pp. 10–11, 2011. 2.
[50]
D. C. Niehorster, L. Li, and M. Lappe. The accuracy and precision of position and orientation tracking in the htc vive virtual reality system for scientific research. i-Perception, 8(3):2041669517708205, 2017. 5.
[51]
S. Noh, H.-S. Yeo, and W. Woo. An hmd-based mixed reality system for avatar-mediated remote collaboration with bare-hand interaction. In ICAT-EGVE, pp. 61–68, 2015. 3.
[52]
R. Otono, N. Isoyama, H. Uchiyama, and K. Kiyokawa. Third-person perspective avatar embodiment in augmented reality: Examining the proteus effect on physical performance. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 730–731. IEEE, 2022. 3.
[53]
T. F. O'Connor, M. E. Fach, R. Miller, S. E. Root, P. P. Mercier, and D. J. Lipomi. The language of glove: Wireless gesture decoder with low-power and stretchable hybrid electronics. PloS one, 12(7):e0179766, 2017. 2.
[54]
W. Piekarski and B. H. Thomas. Interactive augmented reality techniques for construction at a distance of 3d geometry. In Proceedings of the workshop on Virtual environments 2003, pp. 19–28, 2003. 2.
[55]
J. S. Pierce, A. S. Forsberg, M. J. Conway, S. Hong, R. C. Zeleznik, and M. R. Mine. Image plane interaction techniques in 3d immersive environments. In Proceedings of the 1997 symposium on Interactive 3D graphics, pp. 39-ff, 1997. 5.
[56]
T. Piumsomboon, A. Clark, and M. Billinghurst. Physically-based interaction for tabletop augmented reality using a depth-sensing camera for environment mapping. Proc. Image and Vision Computing New Zealand (IVCNZ-2011), pp. 161–166, 2011. 2.
[57]
T. Piumsomboon, A. Clark, M. Billinghurst, and A. Cockburn. User-defined gestures for augmented reality. In IFIP Conference on Human-Computer Interaction, pp. 282–299. Springer, 2013. 2.
[58]
R. Poelman, O. Akman, S. Lukosch, and P. Jonker. As if being there: mediated reality for crime scene investigation. In Proceedings of the ACM 2012 conference on computer supported cooperative work, pp. 1267–1276, 2012. 2.
[59]
M. Prilla, M. Janβen, and T. Kunzendorff. How to interact with augmented reality head mounted devices in care work? a study comparing handheld touch (hands-on) and gesture (hands-free) interaction. AIS Transactions on Human-Computer Interaction, 11(3):157–178, 2019.2.
[60]
M. Quandt, D. Hippert, T. Beinke, and M. Freitag. User-centered evaluation of the learning effects in the use of a 3d gesture control for a mobile location-based augmented reality solution for maintenance. In DELbA@ EC-TEL, 2020. 3.
[61]
R. Radkowski and C. Stritzke. Interactive hand gesture-based assembly for augmented reality applications. In Proceedings of the 2012 International Conference on Advances in Computer-Human Interactions, pp. 303–308. Citeseer, 2012. 2.
[62]
A. Robb, K. Kohm, and J. Porter. Experience matters: Longitudinal changes in sensitivity to rotational gains in virtual reality. ACM Transactions on Applied Perception, 19(4):1–18, 2022. 4.
[63]
N. Rosa. Player/avatar body relations in multimodal augmented reality games. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 550–553, 2016. 3.
[64]
B. Schiettecatte and J. Vanderdonckt. Audiocubes: a distributed cube tangible interface based on interaction range for sound design. In Proceedings of the 2nd international conference on Tangible and embedded interaction, pp. 3–10, 2008. 2.
[65]
S. Schmidt, O. J. A. Nunez, and F. Steinicke. Blended agents: Manipulation of physical objects within mixed reality environments and beyond. In Symposium on Spatial User Interaction, pp. 1–10, 2019. 3.
[66]
R. Serrano, P. Morillo, S. Casas, and C. Cruz-Neira. An empirical evaluation of two natural hand interaction systems in augmented reality. Multi-media Tools and Applications, pp. 1–27, 2022. 3.
[67]
L. E. Sibert and R. J. Jacob. Evaluation of eye gaze interaction. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 281–288, 2000. 2.
[68]
T. A. Snijders and R. J. Bosker. Multilevel analysis: An introduction to basic and advanced multilevel modeling. sage, 2011. 6.
[69]
A. B. Soares, E. A. L. Junior, A. de Oliveira Andrade, and A. Cardoso. Virtual and augmented reality: A new approach to aid users of myoelectric prostheses. Computational Intelligence in Electromyography Analysis-A Perspective on Current Applications and Future Challenges, pp. 409–426, 2012. 3.
[70]
M. Speicher, B. D. Hall, and M. Nebeling. What is mixed reality? In Proceedings of the 2019 CHI conference on human factors in computing systems, pp. 1–15, 2019. 1.
[71]
G. Sroka, L. S. Feldman, M. C. Vassiliou, P. A. Kaneva, R. Fayez, and G. M. Fried. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. The American journal of surgery, 199(1):115–120, 2010. 3, 5.
[72]
R. Terrier, F. Argelaguet, J.-M. Normand, and M. Marchal. Evaluation of ar inconsistencies on ar placement tasks: A vr simulation study. In Virtual Reality and Augmented Reality: 15th Euro Vr International Conference, Euro VR 2018, London, UK, October 22–23, 2018, Proceedings 15, pp. 190–210. Springer, 2018. 8.
[73]
P. P. Valentini. Natural interface for interactive virtual assembly in augmented reality using leap motion controller. International Journal on Interactive Design and Manufacturing (IJIDeM), 12(4):1157–1165, 2018. 2.
[74]
R. Venkatakrishnan, R. Venkatakrishnan, B. Raveendranath, C. C. Pagano, A. C. Robb, W.-C. Lin, and S. V. Babu. Give me a hand: Improving the effectiveness of near-field augmented reality interactions by avatarizing users' end effectors. IEEE Transactions on Visualization and Computer Graphics, 29(5):2412–2422, 2023. 2, 3,4,5,6,8, 9.
[75]
R. Venkatakrishnan, R. Venkatakrishnan, B. Raveendranath, C. C. Pagano, A. C. Robb, W.-C. Lin, and S. V. Babu. How virtual hand representations affect the perceptions of dynamic affordances in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 29(5):2258–2268, 2023. 6.
[76]
R. Y. Wang and J. Popovic. Real-time hand-tracking with a color glove. ACM transactions on graphics (TOG), 28(3):1–8, 2009. 2.
[77]
Wen. Wen 3921 16-inch two-direction variable speed scroll saw, 2023. https://wenproducts.com/products/wen-3921-16-inch-two-direction-variable-speed-scroll-saw. 3.
[78]
M. Whitlock, E. Harnner, J. R. Brubaker, S. Kane, and D. A. Szafir. Interacting with distant objects in augmented reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 41–48. IEEE, 2018. 1,2.
[79]
J. Wither and T. Hollerer. Evaluating techniques for interaction at a distance. In Eighth International Symposium on Wearable Computers, vol. 1, pp. 124–127. IEEE, 2004. 3.
[80]
E. Wolf, N. Dollinger, D. Mal, C. Wienrich, M. Botsch, and M. E. Latoschik. Body weight perception of females using photorealistic avatars in virtual and augmented reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 462–473. IEEE, 2020. 3.
[81]
N. Yee, J. N. Bailenson, and N. Ducheneaut. The proteus effect: Implications of transformed digital self-representation on online and offline behavior. Communication Research, 36(2):285–312, 2009. 9.
[82]
A. Zenner and A. Kruger, Estimating detection thresholds for desktop-scale hand redirection in virtual reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 47–55. IEEE, 2019. 2.
[83]
H. Zhao and W. H. Warren. On-line and model-based approaches to the visual control of action. Vision research, 110:190–202, 2015. 8.
[84]
F. Zhou, H. B.-L. Duh, and M. Billinghurst. Trends in augmented reality tracking, interaction and display: A review of ten years of ismar. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 193–202. IEEE, 2008. 2.
Recommendations
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Mobile Displays for Cross-Reality Interactions between Virtual and Physical Realities
MUM '21: Proceedings of the 20th International Conference on Mobile and Ubiquitous MultimediaWe present two use cases of mobile displays in cross-reality interactions between users immersed in Virtual Reality (VR) and users present in the Physical Reality (PR) by using the mobile display to show select artefacts of interest. The first use case ...
Integrating in-hand physical objects in mixed reality interactions
IUI '22 Companion: Companion Proceedings of the 27th International Conference on Intelligent User InterfacesWith the launch of commercial mixed reality headsets, it has become increasingly important to find new interaction modalities making use of their capabilities. Indeed, endowed with spatial understanding, those devices offer the possibility to interact ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1077-2626 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 04 March 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 19 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in