research-article

Combining Auditory and Mid-Air Haptic Feedback for a Light Switch Button

Authors:

Isa RuttenAuthors Info & Claims

ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

Pages 60 - 69

https://doi.org/10.1145/3382507.3418823

Published: 22 October 2020 Publication History

Abstract

As Mid-Air Haptic (MAH) feedback, which provides a sensation of touch without direct physical contact, is a relatively new technology, research investigating MAH feedback in home usage as well as multi-sensory integration with MAH feedback is still scarce. To address this gap, we propose a possible usage context for MAH feedback, perform an experiment by manipulating auditory and haptic feedback in various physical qualities and suggest possible combinations for positive experiences. Certain sensory combinations led to changes in the emotional responses, as well as the responses regarding utilitarian (e.g. clarity) and perceptual (sensory match) qualities. The results show an added value of MAH feedback when added to sensory compositions, and an increase in the positive experiences induced by MAH length and multimodality.

Supplementary Material

MP4 File (3382507.3418823.mp4)

\r\nIn this paper, we aim to explore user responses towards multi-sensory qualities of a mid-air haptic product. For this, we defined three main steps which start from proposing a usage context for the mid-air haptics device, designing multi-sensory properties and testing user responses to these qualities in an experimental study. We proposed to design a light switch haptic button with various auditory (sound types and frequency) and mid-air haptic qualities (length). According to the results of our experiment, these sensory qualities led to changes in perceived pleasantness, perceived clarity and perceived sensory match.\r\n

Download
79.92 MB

References

[1]

Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology, 505--514.

Digital Library

[2]

Daniel Hajas, Dario Pittera, Antony Nasce, Orestis Georgiou, and Marianna Obrist. 2020. Mid-air haptic rendering of 2D geometric shapes with a dynamic tactile pointer. IEEE Trans. Haptics 13, 1 (2020).

[3]

Damien Ablart, William Frier, Hannah Limerick, Orestis Georgiou, and Marianna Obrist. 2019. Using Ultrasonic Mid-air Haptic Patterns in MultiModal User Experiences. In 2019 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE), IEEE, 1--6.

[4]

William Frier, Damien Ablart, Jamie Chilles, Benjamin Long, Marcello Giordano, Marianna Obrist, and Sriram Subramanian. 2018. Using spatiotemporal modulation to draw tactile patterns in mid-air. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Springer, 270--281.

[5]

Lawrence Van den Bogaert, David Geerts, and Isa Rutten. 2019. Grasping the Future: Identifying Potential Applications for Mid-Air Haptics in the Home. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19), Association for Computing Machinery, Glasgow, Scotland Uk, 1--6.

[6]

Dario Pittera. 2018. Understanding and designing embodied experiences through mid-air tactile stimulation. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, 709--711.

Digital Library

[7]

Dario Pittera, Damien Ablart, and Marianna Obrist. 2019. Creating an illusion of movement between the hands using mid-air touch. IEEE Trans. Haptics 12, 4 (2019), 615--623.

Digital Library

[8]

Orestis Georgiou, Hannah Limerick, Loïc Corenthy, Mark Perry, Mykola Maksymenko, Sam Frish, Jörg Müller, Myroslav Bachynskyi, and Jin Ryong Kim. 2019. Mid-Air Haptic Interfaces for Interactive Digital Signage and Kiosks. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1--9.

[9]

Damien Ablart, Carlos Velasco, and Marianna Obrist. 2017. Integrating mid-air haptics into movie experiences. In Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video, 77--84.

Digital Library

[10]

Loic Corenthy, Marcello Giordano, Richard Hayden, Daniel Griffiths, Craig Jeffrey, Hannah Limerick, Orestis Georgiou, Tom Carter, Jörg Müller, and Sriram Subramanian. 2018. Touchless tactile displays for digital signage: midair haptics meets large screens. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 1--4.

Digital Library

[11]

Hannah Limerick, Richard Hayden, David Beattie, Orestis Georgiou, and Jörg Müller. 2019. User engagement for mid-air haptic interactions with digital signage. In Proceedings of the 8th ACM International Symposium on Pervasive Displays, 1--7.

Digital Library

[12]

Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist. 2017. Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. Int. J. Hum.-Comput. Stud. 108, (December 2017), 1--14.

Digital Library

[13]

Ted Romanus, Sam Frish, Mykola Maksymenko, William Frier, Loïc Corenthy, and Orestis Georgiou. 2020. Mid-Air Haptic Bio-Holograms in Mixed Reality. ArXiv Prepr. ArXiv200101441 (2020).

[14]

HENDRIK N. J. Schifferstein and CHARLES Spence. 2008. 5 MULTISENSORY PRODUCT EXPERIENCE. In Product Experience, Hendrik N. J. Schifferstein and Paul Hekkert (eds.). Elsevier, San Diego, 133-- 161.

[15]

Klemens Knöferle and Charles Spence. 2012. Crossmodal correspondences between sounds and tastes. Psychon. Bull. Rev. (2012), 1--15.

[16]

Elif Özcan, Gerald C. Cupchik, and Hendrik NJ Schifferstein. 2017. Auditory and visual contributions to affective product quality. Int. J. Des. 11, 1 (2017), 35--50.

[17]

Hendrik NJ Schifferstein and Pieter MA Desmet. 2008. Tools facilitating multisensory product design. Des. J. 11, 2 (2008), 137--158.

[18]

Hendrik N. J. Schifferstein. 2011. Multi sensory design. In Procedings of the Second Conference on Creativity and Innovation in Design - DESIRE '11, ACM Press, Eindhoven, Netherlands, 361.

Digital Library

[19]

Charles Spence. 2012. Managing sensory expectations concerning products and brands: Capitalizing on the potential of sound and shape symbolism. J. Consum. Psychol. 22, 1 (January 2012), 37--54.

[20]

Graham Wilson, Thomas Carter, Sriram Subramanian, and Stephen A. Brewster. 2014. Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14), Association for Computing Machinery, Toronto, Ontario, Canada, 1133--1142.

Digital Library

[21]

Benjamin Long, Sue Ann Seah, Tom Carter, and Sriram Subramanian. 2014. Rendering volumetric haptic shapes in mid-air using ultrasound. ACM Trans. Graph. TOG 33, 6 (2014), 1--10.

Digital Library

[22]

Antti Sand, Ismo Rakkolainen, Poika Isokoski, Jari Kangas, Roope Raisamo, and Karri Palovuori. 2015. Head-mounted display with mid-air tactile feedback. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology, 51--58.

Digital Library

[23]

Gözel Shakeri, John H. Williamson, and Stephen Brewster. 2018. May the Force Be with You: Ultrasound Haptic Feedback for Mid-Air Gesture Interaction in Cars. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '18), Association for Computing Machinery, Toronto, ON, Canada, 1--10.

Digital Library

[24]

Elif Özcan. 2014. The Harley effect: Internal and external factors that facilitate positive experiences with product sounds. J. Sonic Stud. 6, 1 (2014), a07.

[25]

Rick Schifferstein, Jacco Otten, Fien Thoolen, and Paul Hekkert. 2010. The experimental assessment of sensory dominance in a product development context. J Des. Res. 8, (January 2010).

[26]

Carlos Velasco and Charles Spence. 2019. The Multisensory Analysis of Product Packaging Framework. Multisensory Packag. (2019), 191--223.

[27]

Elif Özcan and Hendrik N J Schifferstein. THE EFFECT OF (UN)PLEASANT SOUNDS ON THE VISUAL AND OVERALL PLEASANTNESS OF PRODUCTS. 7.

[28]

Anna Fenko, Hendrik NJ Schifferstein, and Paul Hekkert. 2010. Shifts in sensory dominance between various stages of user--product interactions. Appl. Ergon. 41, 1 (2010), 34--40.

[29]

Richard H. Lyon. 2003. Product sound quality-from perception to design. Sound Vib. 37, 3 (2003), 18--23.

[30]

Marianna Obrist, Carlos Velasco, Chi Thanh Vi, Nimesha Ranasinghe, Ali Israr, Adrian D. Cheok, Charles Spence, and Ponnampalam Gopalakrishnakone. 2016. Touch, taste, & smell user interfaces: The future of multisensory HCI. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 3285--3292.

Digital Library

[31]

Yeongmi Kim, Jongeun Cha, Ian Oakley, and Jeha Ryu. 2009. Exploring tactile movies: An initial tactile glove design and concept evaluation. Ieee Multimed. (2009).

[32]

Paul Lemmens, Floris Crompvoets, Dirk Brokken, Jack Van Den Eerenbeemd, and Gert-Jan de Vries. 2009. A body-conforming tactile jacket to enrich movie viewing. In World Haptics 2009-Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, IEEE, 7--12.

Digital Library

[33]

Inwook Hwang, Hyungki Son, and Jin Ryong Kim. 2017. AirPiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback. In 2017 IEEE World Haptics Conference (WHC), 213--218.

[34]

Emanuela Maggioni, Erika Agostinelli, and Marianna Obrist. 2017. Measuring the added value of haptic feedback. In 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), IEEE, 1--6.

[35]

Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2014. Tactile feedback for above-device gesture interfaces: Adding touch to touchless interactions. In Proceedings of the 16th International Conference on Multimodal Interaction, 419--426.

Digital Library

[36]

Isa Rutten, William Frier, Lawrence Van den Bogaert, and David Geerts. 2019. Invisible Touch: How Identifiable are Mid-Air Haptic Shapes? In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1--6.

Digital Library

[37]

Hugo Fastl. 2005. Psycho-Acoustics and Sound Quality. In Communication Acoustics, Jens Blauert (ed.). Springer, Berlin, Heidelberg, 139--162.

[38]

M. Zampini, S. Guest, and C. Spence. 2003. The Role of Auditory Cues in Modulating the Perception of Electric Toothbrushes. J. Dent. Res. 82, 11 (November 2003), 929--932.

[39]

Teah-Marie Bynion and Matthew T. Feldner. 2017. Self-Assessment Manikin. In Encyclopedia of Personality and Individual Differences, Virgil Zeigler-Hill and Todd K. Shackelford (eds.). Springer International Publishing, Cham, 1--3.

[40]

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: the selfassessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25, 1 (1994), 49--59.

[41]

James A. Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. J. Res. Personal. 11, 3 (1977), 273--294.

[42]

Gaël François Gérard Laurans. 2011. On the moment-to-moment measurement of emotion during person-product interaction: By means of video-supported retrospective self-report, with some ancillary remarks on other issues in designrelated emotion measurement. (2011).

[43]

Gaël Laurans and Pieter Desmet. NEW DIRECTIONS FOR THE NONVERBAL MEASUREMENT OF EMOTION IN DESIGN. 13.

[44]

Ivan Poupyrev and Shigeaki Maruyama. 2003. Tactile interfaces for small touch screens. In Proceedings of the 16th annual ACM symposium on User interface software and technology, 217--220.

Digital Library

[45]

Graham Wilson, Thomas Carter, Sriram Subramanian, and Stephen A. Brewster. 2014. Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14), Association for Computing Machinery, Toronto, Ontario, Canada, 1133--1142.

Digital Library

[46]

Sonja Rümelin, Thomas Gabler, and Jesper Bellenbaum. 2017. Clicks are in the Air: How to Support the Interaction with Floating Objects through Ultrasonic Feedback. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ACM, Oldenburg Germany, 103--108.

Digital Library

[47]

Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions Mediated Through Mid-Air Haptics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems CHI '15, ACM Press, Seoul, Republic of Korea, 2053--2062.

Digital Library

[48]

A. Donald. 2004. Norman. Emotional Design: Why We Love (or Hate) Everyday Things. (2004).

[49]

Erdem Demir, Pieter M A Desmet, and Paul Hekkert. Appraisal Patterns of Emotions in User-Product Interaction. 15.

[50]

Pieter Desmet and Eva Dijkhuis. A Wheelchair can be Fun: A Case of Emotiondriven Design. 6.

[51]

Patrick W. Jordan. 2002. Designing Pleasurable Products: An Introduction to the New Human Factors. CRC Press.

[52]

Lau Langeveld, René van Egmond, Reinier Jansen, and Elif Özcan. 2013. Product Sound Design: Intentional and Consequential Sounds. Adv. Ind. Des. Eng. (March 2013).

[53]

Elia Gatti, Elena Calzolari, Emanuela Maggioni, and Marianna Obrist. 2018. Emotional ratings and skin conductance response to visual, auditory and haptic stimuli. Sci. Data 5, 1 (December 2018), 180120.

Cited By

Xue JMontano Murillo RDawes CFrier WCornelio PObrist M(2024)FabSound: Audio-Tactile and Affective Fabric Experiences Through Mid-air HapticsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642533(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642533
Montano-Murillo RPittera DFrier WGeorgiou OObrist MCornelio P(2024)It Sounds Cool: Exploring Sonification of Mid-Air Haptic Textures Exploration on Texture Judgments, Body Perception, and Motor BehaviourIEEE Transactions on Haptics10.1109/TOH.2023.332049217:2(237-248)Online publication date: Apr-2024
https://doi.org/10.1109/TOH.2023.3320492
Wojna KGeorgiou OBeattie DFrier WWright MLutteroth C(2023)An Exploration of Just Noticeable Differences in Mid-Air Haptics2023 IEEE World Haptics Conference (WHC)10.1109/WHC56415.2023.10224388(410-416)Online publication date: 10-Jul-2023
https://doi.org/10.1109/WHC56415.2023.10224388
Show More Cited By

Recommendations

Haptic and Auditory Feedback on Immersive Media in Virtual Reality
VRST '24: Proceedings of the 30th ACM Symposium on Virtual Reality Software and Technology

In Virtual Reality (VR), visual and auditory sensations are effectively leveraged to create immersive experiences. However, touch is significantly underutilized in immersive media. We enhance the VR image viewing experience by integrating haptic and ...
Head-mounted display with mid-air tactile feedback
VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

Virtual and physical worlds are merging. Currently users of head-mounted displays cannot have unobtrusive tactile feedback while touching virtual objects. We present a mid-air tactile feedback system for head-mounted displays. Our prototype uses the ...
Light-weight immaterial particle displays with mid-air tactile feedback
2015 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE)
Immaterial mid-air displays formed of flowing light-scattering particles are becoming feasible for displaying information in thin air and interacting with it. With light-weight desktop fogscreens and low-cost hand tracking, the user can easily and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

October 2020

920 pages

ISBN:9781450375818

DOI:10.1145/3382507

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Imec-VLAIO

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
394
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xue JMontano Murillo RDawes CFrier WCornelio PObrist M(2024)FabSound: Audio-Tactile and Affective Fabric Experiences Through Mid-air HapticsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642533(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642533
Montano-Murillo RPittera DFrier WGeorgiou OObrist MCornelio P(2024)It Sounds Cool: Exploring Sonification of Mid-Air Haptic Textures Exploration on Texture Judgments, Body Perception, and Motor BehaviourIEEE Transactions on Haptics10.1109/TOH.2023.332049217:2(237-248)Online publication date: Apr-2024
https://doi.org/10.1109/TOH.2023.3320492
Wojna KGeorgiou OBeattie DFrier WWright MLutteroth C(2023)An Exploration of Just Noticeable Differences in Mid-Air Haptics2023 IEEE World Haptics Conference (WHC)10.1109/WHC56415.2023.10224388(410-416)Online publication date: 10-Jul-2023
https://doi.org/10.1109/WHC56415.2023.10224388
Dube TRen YLimerick HMacKenzie IArif A(2022)Push, Tap, Dwell, and Pinch: Evaluation of Four Mid-air Selection Methods Augmented with Ultrasonic Haptic FeedbackProceedings of the ACM on Human-Computer Interaction10.1145/35677186:ISS(207-225)Online publication date: 14-Nov-2022
https://dl.acm.org/doi/10.1145/3567718
Morales RPittera DGeorgiou OKappus BFrier W(2022)UltraButton: A Minimalist Touchless Multimodal Haptic ButtonIEEE Transactions on Haptics10.1109/TOH.2022.321432215:4(729-740)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TOH.2022.3214322
Maunsbach MHornbæk KSeifi H(2022)Whole-Hand Haptics for Mid-air ButtonsHaptics: Science, Technology, Applications10.1007/978-3-031-06249-0_33(292-300)Online publication date: 22-May-2022
https://dl.acm.org/doi/10.1007/978-3-031-06249-0_33
Kim J(2022)Multimodal Interaction with Mid-Air HapticsUltrasound Mid-Air Haptics for Touchless Interfaces10.1007/978-3-031-04043-6_7(185-205)Online publication date: 17-Sep-2022
https://doi.org/10.1007/978-3-031-04043-6_7
Cornelio P(2022)Opportunities for Multisensory Mid-Air Interactions Featuring Ultrasound Haptic FeedbackUltrasound Mid-Air Haptics for Touchless Interfaces10.1007/978-3-031-04043-6_4(99-117)Online publication date: 17-Sep-2022
https://doi.org/10.1007/978-3-031-04043-6_4
Georgiou OFrier WSchneider O(2022)User Experience and Mid-Air Haptics: Applications, Methods, and ChallengesUltrasound Mid-Air Haptics for Touchless Interfaces10.1007/978-3-031-04043-6_2(21-69)Online publication date: 17-Sep-2022
https://doi.org/10.1007/978-3-031-04043-6_2
Georgiou OFrier WFreeman EPacchierotti CHoshi T(2022)Mid-Air Haptics: Future Challenges and OpportunitiesUltrasound Mid-Air Haptics for Touchless Interfaces10.1007/978-3-031-04043-6_18(385-397)Online publication date: 17-Sep-2022
https://doi.org/10.1007/978-3-031-04043-6_18
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten