poster

Tactile Tone System: A Wearable Device to Assist Accuracy of Vocal Pitch in Cochlear Implant Users

Authors:

Hyungcheol ShinAuthors Info & Claims

ASSETS '20: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility

Article No.: 81, Pages 1 - 3

https://doi.org/10.1145/3373625.3418008

Published: 29 October 2020 Publication History

Abstract

Cochlear implantation is an effective tool in speech perception. However, activities such as listening to music and singing remain challenging for cochlear implant (CI) users, due to inaccurate pitch recognition. In this study, we propose a method for CI users to recognize precise pitch differences through tactile feedback. The proposed system encodes real-time audio signals to 36 musical tones (from C3 to B5), represented by tactile codes using nine vibration motors in a glove-type device. Two CI users participated in 15 h of training using our system and showed significant improvement in pitch accuracy while singing. In addition to the quantitative results, both participants expressed satisfaction in distinguishing and vocalizing musical tones, which led to increased interest in music. This study provides opportunities for CI users to engage more deeply and participate in musical education as well as achieve improved aural rehabilitation.

References

[1]

Keva Abotomey. 2008. Music in the Lives of Deaf Students in an Australian School. University of Sydney Honours Theses - Music Education.

[2]

Paul Bach-Y-Rita, Carter C. Collins, Frank A. Saunders, Benjamin White, and Lawrence Scadden. 1969. Vision Substitution by Tactile Image Projection. Nature 221(1969), 963–964. https://doi.org/10.1038/221963a0

[3]

Kate Gfeller, Shelley Witt, Linda Spencer, Julie Stordahl, and J. Tomblin. 1998. Musical Involvement and Enjoyment of Children Who Use Cochlear Implants. Volta Review 100 (12 1998).

[4]

Juan Huang, Benjamin Sheffield, Payton Lin, and Fan-Gang Zeng. 2017. Electro-Tactile Stimulation Enhances Cochlear Implant Speech Recognition in Noise. Scientific Reports 7, 2196 (2017). https://doi.org/10.1038/s41598-017-02429-1

[5]

Kevin Huang, Thad Starner, Ellen Do, Gil Weinberg, Daniel Kohlsdorf, Claas Ahlrichs, and Ruediger Leibrandt. 2010. Mobile Music Touch: Mobile Tactile Stimulation for Passive Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). ACM, New York, NY, USA, 791–800. https://doi.org/10.1145/1753326.1753443

Digital Library

[6]

Robert Kang, Grace Liu Nimmons, Ward Drennan, Jeff Longnion, Chad Ruffin, Kaibao Nie, Jong Won, Tina Worman, Bevan Yueh, and Jay Rubinstein. 2009. Development and Validation of the University of Washington Clinical Assessment of Music Perception Test. Ear and hearing 30(2009), 411–8. https://doi.org/10.1097/AUD.0b013e3181a61bc0

[7]

Valerie Looi, Hugh McDermott, Colette McKay, and Louise Hickson. 2004. Pitch discrimination and melody recognition by cochlear implant users. International Congress Series 1273 (2004), 197 – 200. https://doi.org/10.1016/j.ics.2004.08.038

[8]

Granit Luzhnica and Eduardo Veas. 2019. Optimising Encoding for Vibrotactile Skin Reading. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 235, 14 pages. https://doi.org/10.1145/3290605.3300465

Digital Library

[9]

Antonella Mazzoni and Nick Bryan-Kinnsb. 2016. Mood Glove: A haptic wearable prototype system to enhance mood music in film. Entertainment Computing 17 (2016), 9–17. https://doi.org/10.1016/j.entcom.2016.06.002

[10]

Hugh J McDermott. 2004. Music perception with cochlear implants: a review. Trends in amplification 8, 2 (2004), 49–82. https://doi.org/10.1177/108471380400800203

[11]

Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, and S H. Ong. 2009. An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). ACM, New York, NY, USA, 337–346. https://doi.org/10.1145/1518701.1518756

Digital Library

[12]

Scott David Novich. 2015. Sound-to-Touch Sensory Substitution and Beyond. Ph.D. Dissertation. Rice University, Houston, TX.

[13]

Francoise Rochette, Aline Moussard, and Emmanuel Bigand. 2014. Music lessons improve auditory perceptual and cognitive performance in deaf children. Frontiers in human neuroscience 8, 488 (2014). https://doi.org/10.3389/fnhum.2014.00488

[14]

M. Sakajiri, S. Miyoshi, K. Nakamura, S. Fukushima, and T. Ifukube. 2013. Accuracy of Voice Pitch Control in Singing Using Tactile Voice Pitch Feedback Display. In 2013 IEEE International Conference on Systems, Man, and Cybernetics. 4201–4206. https://doi.org/10.1109/SMC.2013.716

Digital Library

[15]

Ritva Torppa, Andrew Faulkner, Teija Kujala, Minna Huotilainen, and Jari Lipsanen. 2018. Developmental Links Between Speech Perception in Noise, Singing, and Cortical Processing of Music in Children with Cochlear Implants. Music Perception: An Interdisciplinary Journal 36, 2 (2018), 156–174. https://doi.org/10.1525/mp.2018.36.2.156 arXiv:https://mp.ucpress.edu/content/36/2/156.full.pdf

[16]

Lei Xu, Ning Zhou, Xiuwu Chen, Yongxin Li, and Demin Han. 2009. Vocal singing by prelingually-deafened children with cochlear implants. Hearing Research 255(2009), 129–134. https://doi.org/10.1016/j.heares.2009.06.011

[17]

Fan-Gang Zeng, Qing Tang, and Thomas Lu. 2014. Abnormal Pitch Perception Produced by Cochlear Implant Stimulation. PloS one 9 (02 2014), e88662. https://doi.org/10.1371/journal.pone.0088662

Cited By

Jeong CSong YShin SKim M(2024)Efficient pitch‐estimation network for edge devicesETRI Journal10.4218/etrij.2023-0430Online publication date: 2-May-2024
https://doi.org/10.4218/etrij.2023-0430
Choi YJeon JLee CNoh YHong J(2024)A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music ConceptsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642665(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642665
Flores Ramones Adel-Rio-Guerra M(2023)Recent Developments in Haptic Devices Designed for Hearing-Impaired People: A Literature ReviewSensors10.3390/s2306296823:6(2968)Online publication date: 9-Mar-2023
https://doi.org/10.3390/s23062968
Show More Cited By

Index Terms

Tactile Tone System: A Wearable Device to Assist Accuracy of Vocal Pitch in Cochlear Implant Users
1. Applied computing
  1. Life and medical sciences
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Haptic devices

Index terms have been assigned to the content through auto-classification.

Recommendations

Why People with a Cochlear Implant Listen to Music
Perception, Representations, Image, Sound, Music
Abstract
The cochlear implant (CI) is the most successful neural prosthetic device in the market. It allows hundreds of thousands of people around the world to regain a sense of hearing. However, unlike a pair of glasses that can restore vision perfectly, ...
Accuracy of Voice Pitch Control in Singing Using Tactile Voice Pitch Feedback Display
SMC '13: Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics

Deaf blind and hearing impaired persons cannot perceive their own voice pitch and thus find it difficult to control. While singing, the voice pitch needs to be controlled to maintain a stable tone. Hence, a voice pitch control system was developed to ...
On the Recognition of Cochlear Implant-Like Spectrally Reduced Speech With MFCC and HMM-Based ASR

This correspondence investigates the recognition of cochlear implant-like spectrally reduced speech (SRS) using mel frequency cepstral coefficient (MFCC) and hidden Markov model (HMM)-based automatic speech recognition (ASR). The SRS was synthesized ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '20: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility

October 2020

764 pages

ISBN:9781450371032

DOI:10.1145/3373625

Editors:
Tiago Guerreiro
University of Lisbon, Portugal
,
Hugo Nicolau
INESC-ID, Portugal
,
Karyn Moffatt
McGill University, Canada

Copyright © 2020 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2020

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

ASSETS '20

Sponsor:

SIGACCESS

ASSETS '20: The 22nd International ACM SIGACCESS Conference on Computers and Accessibility

October 26 - 28, 2020

Virtual Event, Greece

Acceptance Rates

ASSETS '20 Paper Acceptance Rate 46 of 167 submissions, 28%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
223
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jeong CSong YShin SKim M(2024)Efficient pitch‐estimation network for edge devicesETRI Journal10.4218/etrij.2023-0430Online publication date: 2-May-2024
https://doi.org/10.4218/etrij.2023-0430
Choi YJeon JLee CNoh YHong J(2024)A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music ConceptsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642665(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642665
Flores Ramones Adel-Rio-Guerra M(2023)Recent Developments in Haptic Devices Designed for Hearing-Impaired People: A Literature ReviewSensors10.3390/s2306296823:6(2968)Online publication date: 9-Mar-2023
https://doi.org/10.3390/s23062968
Ganis FGulli AFontana FSerafin S(2023)The Role of Haptics in Training and Games for Hearing-Impaired Individuals: A Systematic ReviewMultimodal Technologies and Interaction10.3390/mti80100018:1(1)Online publication date: 22-Dec-2023
https://doi.org/10.3390/mti8010001
Paisa RNilsson NSerafin S(2023)Tactile displays for auditory augmentation–A scoping review and reflections on music applications for hearing impaired usersFrontiers in Computer Science10.3389/fcomp.2023.10855395Online publication date: 27-Apr-2023
https://doi.org/10.3389/fcomp.2023.1085539
Fletcher M(2021)Can Haptic Stimulation Enhance Music Perception in Hearing-Impaired Listeners?Frontiers in Neuroscience10.3389/fnins.2021.72387715Online publication date: 31-Aug-2021
https://doi.org/10.3389/fnins.2021.723877

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents