Work in Progress

Grasping Temperature: Thermal Feedback in VR Robot Teleoperation

Authors:

Leonor Fermoselle,

Alexander Toet,

Jeanine van Bruggen,

Nanda van der Stap,

Frank B. ter Haar,

Jan Van ErpAuthors Info & Claims

IMX '22: Proceedings of the 2022 ACM International Conference on Interactive Media Experiences

Pages 261 - 266

https://doi.org/10.1145/3505284.3532969

Published: 22 June 2022 Publication History

Abstract

This paper presents a proof-of-concept of a robotic teleoperation system, that provides the human operator a thermal sense in addition to the visual sense. With a sensor suite comprising a stereo camera, 360° camera and long-wave infra-red camera, our demonstrator pushes the boundaries of virtual-reality situational awareness by bringing not only 3D visual content but also a 360° thermal experience to the operator. The visual channel of our robotic teleoperation system is represented through a head-mounted-display and the thermal channel is displayed through directional heaters in the operator cockpit and a thermal glove. Initial tests showed that an operator successfully experienced a 360° remote environment, correctly distinguished between and interacted with hot and cold objects, and could notice the presence of nearby people outside her direct field-of-view, based on their emitted heat.

References

[1]

Jonathan Boisvert, Marc-Antoine Drouin, Guy Godin, and Michel Picard. 2020. Augmented reality, 3D measurement, and thermal imagery for computer-assisted manufacturing. In Emerging Digital Micromirror Device Based Systems and Applications XII, Vol. 11294. International Society for Optics and Photonics, SPIE Digital Library, United States, 112940L.

[2]

Carlos Coelho, JG Tichon, Trevor J Hine, GM Wallis, and Giuseppe Riva. 2006. Media presence and inner presence: the sense of presence in virtual reality technologies. In From communication to presence: Cognition, emotions and culture towards the ultimate communicative experience. IOS Press, Amsterdam, Amsterdam, Netherlands, 25–45.

[3]

I. Comsa, R. Trestian, and G. Ghinea. 2018. 360° Mulsemedia experience over next generation wireless networks – A reinforcement learning approach. In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, United States, 1–6. https://doi.org/10.1109/QoMEX.2018.8463409

[4]

I. Comşa, E. B. Saleme, A. Covaci, G. M. Assres, R. Trestian, C. A. S. Santos, and G. Ghinea. 2020. Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience. IEEE MultiMedia 27, 1 (2020), 27–36. https://doi.org/10.1109/MMUL.2019.2954405

[5]

Richard L. Daft and Robert H. Lengel. 1986. Organizational information requirements, media richness and structural design. Management Science 32, 5 (1986), 554–571. https://doi.org/10.1287/mnsc.32.5.554

Digital Library

[6]

Richard L Daft, Robert H Lengel, and Linda Klebe Trevino. 1987. Message equivocality, media selection, and manager performance: Implications for information systems. MIS quarterly 11(1987), 355–366.

[7]

Leonor Fermoselle, Simon Gunkel, Frank ter ter Haar, Sylvie Dijkstra-Soudarissanane, Alexander Toet, Omar Niamut, and Nanda van van der Stap. 2020. Let’s Get in Touch! Adding Haptics to Social VR. In ACM International Conference on Interactive Media Experiences. ACM, New York, NY, USA, 174–179.

Digital Library

[8]

FLIR. 2022. FLIR A655sc High-Resolution Science Grade LWIR Camera. https://www.flir.eu/products/a655sc/ Accessed: 2022-03-09.

[9]

Stereolabs Inc.2022. ZED 2i – Industrial AI Stereo Camera. https://www.stereolabs.com/zed-2i/ Accessed: 2022-03-09.

[10]

Seung-Won Kim, Sung Hee Kim, Choong Sun Kim, Kyoungsoo Yi, Jun-Sik Kim, Byung Jin Cho, and Youngsu Cha. 2020. Thermal display glove for interacting with virtual reality. Scientific Reports 10, 1 (2020), 1–12.

[11]

Ricoh Company Ltd.2022. Ricoh Theta Z1. https://theta360.com/en/about/theta/z1.html Accessed: 2022-03-09.

[12]

Honghao Lv, Geng Yang, Huiying Zhou, Xiaoyan Huang, Huayong Yang, and Zhibo Pang. 2020. Teleoperation of collaborative robot for remote dementia care in home environments. IEEE Journal of Translational Engineering in Health and Medicine 8 (2020), 1–10. https://doi.org/10.1109/JTEHM.2020.3002384

[13]

Günter Niemeyer, Carsten Preusche, Stefano Stramigioli, and Dongjun Lee. 2016. Telerobotics. Springer International Publishing, Cham, Switzerland, 1085–1108. https://doi.org/10.1007/978-3-319-32552-1_43

[14]

Kaushik Parida, Hyunwoo Bark, and Pooi See Lee. 2021. Emerging thermal technology enabled augmented reality. Advanced Functional Materials 31, 39 (2021), 2007952.

[15]

Denis Peña and Fumihide Tanaka. 2020. Human perception of social robot’s emotional states via facial and thermal expressions. ACM Transactions on Human-Robot Interaction 9, 4 (2020), Article 26. https://doi.org/10.1145/3388469

Digital Library

[16]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement.

[17]

Halodi Robotics. 2022. Eve Robot. https://www.halodi.com/ Accessed: 2022-03-09.

[18]

Sensiks. 2022. Sensiks Sensory Reality Pods & Platform. https://www.sensiks.com Accessed: 2022-03-09.

[19]

Kazuaki Tanaka, Hideyuki Nakanishi, and Hiroshi Ishiguro. 2015. Physical embodiment can produce robot operator’s pseudo presence. Frontiers in ICT 2, 8 (2015), 1–12. https://doi.org/10.3389/fict.2015.00008

[20]

Unity Technologies. 2022. Unity Real-time Development Platform. https://unity.com/ Accessed: 2022-03-09.

[21]

TEGway. 2022. ThermoReal, TEGway. http://tegway.co/tegway/ Accessed: 2022-03-09.

[22]

©FLIR. 2018. Try the FLIR Virtual Reality Experience!https://www.flir.in/news-center/corporate-news/try-the-flir-virtual-reality-experience/ Accessed: 2021-08-03.

[23]

Alexander Toet, Irene A. Kuling, Bouke N. Krom, and Jan B. F. van Erp. 2020. Toward enhanced teleoperation through embodiment. Frontiers in Robotics and AI 7, 14 (2020), 1–22. https://doi.org/10.3389/frobt.2020.00014

[24]

ETH Zurich. 2022. ALMA – Robotic Systems Lab, ETH Zurich. https://rsl.ethz.ch/robots-media/alma.html Accessed: 2022-03-09.

Cited By

Park WKorres GJamil MEid M(2024)Neural correlates of thermal stimulation during active touchFrontiers in Neuroscience10.3389/fnins.2023.132041717Online publication date: 8-Jan-2024
https://doi.org/10.3389/fnins.2023.1320417
Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Hauser KWatson EBae JBankston JBehnke SBorgia BCatalano MDafarra Svan Erp JFerris TFishel JHoffman GIvaldi SKanehiro FKheddar ALannuzel GMorie JNaughton PNGuyen SOh PPadir TPippine JPark JVaz JPucci DWhitney PWu PLocke D(2024)Analysis and Perspectives on the ANA Avatar XPRIZE CompetitionInternational Journal of Social Robotics10.1007/s12369-023-01095-wOnline publication date: 30-Jan-2024
https://doi.org/10.1007/s12369-023-01095-w

Recommendations

VR-Based Teleoperation for Robot Compliance Control

Robots governed by remote human operators are excellent candidates for work in hazardous or uncertain environments such as nuclear plants or outer space. For successful teleoperation, it is important to let the operator feel physically present at the ...
Virtual and Augmented Reality with Head-Tracking for Efficient Teleoperation of Groups of Robots
CW '08: Proceedings of the 2008 International Conference on Cyberworlds

This paper deals with the usage of Virtual and Augmented Reality techniques in the field of teleoperation. The global context is a project for collaborative teleoperation of real robots groups. The aim is to enable to a group of teleoprators to control ...
A Real-Time Reconfigurable Collision Avoidance System for Robot Manipulation
ICMRE 2017: Proceedings of the 3rd International Conference on Mechatronics and Robotics Engineering

Intelligent robotic systems are becoming fundamental actors in industrial and hazardous facilities scenarios. Aiming to increase personnel safety and machine availability, robots can help perform repetitive and dangerous tasks which humans either prefer ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IMX '22: Proceedings of the 2022 ACM International Conference on Interactive Media Experiences

June 2022

390 pages

ISBN:9781450392129

DOI:10.1145/3505284

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2022

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Conference

IMX '22

Sponsor:

IMX '22: ACM International Conference on Interactive Media Experiences

June 22 - 24, 2022

JB, Aveiro, Portugal

Acceptance Rates

Overall Acceptance Rate 69 of 245 submissions, 28%

Upcoming Conference

IMX '25

Sponsor:
sigchi

ACM International Conference on Interactive Media Experiences

June 3 - 6, 2025

Niter?i , Brazil

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
213
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)2

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Park WKorres GJamil MEid M(2024)Neural correlates of thermal stimulation during active touchFrontiers in Neuroscience10.3389/fnins.2023.132041717Online publication date: 8-Jan-2024
https://doi.org/10.3389/fnins.2023.1320417
Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Hauser KWatson EBae JBankston JBehnke SBorgia BCatalano MDafarra Svan Erp JFerris TFishel JHoffman GIvaldi SKanehiro FKheddar ALannuzel GMorie JNaughton PNGuyen SOh PPadir TPippine JPark JVaz JPucci DWhitney PWu PLocke D(2024)Analysis and Perspectives on the ANA Avatar XPRIZE CompetitionInternational Journal of Social Robotics10.1007/s12369-023-01095-wOnline publication date: 30-Jan-2024
https://doi.org/10.1007/s12369-023-01095-w

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents