Automatic Video-to-Audiotactile Conversion of Golf Broadcasting on A Refreshable Pin Array
Article No.: 15, Pages 1 - 8
Abstract
Video accessibility is an important but challenging research question. In this study, we implemented and evaluated a system that converts video content into audio clips and tactile icons without losing context using a refreshable pin array display. The suggested system converts contextual information of the video to audio description and tactile scenes, allowing users to hear and touch. As an initial target, we selected golf broadcasting, a popular sport in the BLVI community, which has a clear context yet relies heavily on visual features and provides limited information through audio. We extracted contextual information through computer vision to deliver information such as scores and the trajectories and results of shots. Then, we converted them to audio via Text-to-Speech and tactile icons on the pin array. We evaluated the system by conducting a perception experiment and a usability survey, and the results showed that the system effectively converted the information.
Supplemental Material
MP4 File
Supplementary video
- Download
- 50.24 MB
References
[1]
[n. d.]. eBRF and multiline braille displays. https://www.perkins.org/resource/ebrf-and-multiline-braille-displays/
[2]
Nayyer Aafaq, Ajmal Mian, Wei Liu, Syed Zulqarnain Gilani, and Mubarak Shah. 2019. Video description: A survey of methods, datasets, and evaluation metrics. ACM Computing Surveys (CSUR) 52, 6 (2019), 1–37.
[3]
NV Access. 2023. NVDA screen reader. https://www.nvaccess.org/download/
[4]
Chieko Asakawa, Hironobu Takagi, Shuichi Ino, and Tohru Ifukube. 2002. Auditory and tactile interfaces for representing the visual effects on the web. In Proceedings of the fifth international ACM conference on Assistive technologies. 65–72.
[5]
United States Golf Association. 2023. Rules For Golfers with Visual Impaired. https://www.usga.org/rules-hub/rules-for-golfers-with-disabilities/blind-golfers-697de898.html
[6]
Jens Bornschein, Denise Bornschein, and Gerhard Weber. 2018. Blind pictionary: drawing application for blind users. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–4.
[7]
Chongyan Chen, Samreen Anjum, and Danna Gurari. 2023. VQA Therapy: Exploring Answer Differences by Visually Grounding Answers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 15315–15325.
[8]
Giovanni Fusco and Valerie S Morash. 2015. The tactile graphics helper: providing audio clarification for tactile graphics using machine vision. In Proceedings of the 17th international ACM SIGACCESS conference on computers & accessibility. 97–106.
[9]
Haruna Fushimi, Daiya Kato, Youichi Kamiyama, Kazuya Yanagihara, Kouta Minamizawa, and Kai Kunze. 2017. atmoSphere: designing cross-modal music experiences using spatial audio with haptic feedback. In ACM SIGGRAPH 2017 Emerging Technologies. 1–2.
[10]
Giles Hamilton-Fletcher, Marianna Obrist, Phil Watten, Michele Mengucci, and Jamie Ward. 2016. " I Always Wanted to See the Night Sky" Blind User Preferences for Sensory Substitution Devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2162–2174.
[11]
Susumu Harada, Daisuke Sato, Dustin W. Adams, Sri Kurniawan, Hironobu Takagi, and Chieko Asakawa. 2013. Accessible photo album: enhancing the photo sharing experience for people with visual impairment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2127–2136. https://doi.org/10.1145/2470654.2481292
[12]
Leona Holloway, Swamy Ananthanarayan, Matthew Butler, Madhuka Thisuri De Silva, Kirsten Ellis, Cagatay Goncu, Kate Stephens, and Kim Marriott. 2022. Animations at Your Fingertips: Using a Refreshable Tactile Display to Convey Motion Graphics for People who are Blind or have Low Vision. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–16.
[13]
Rankpong Kittinaradorn. 2023. Easy OCR. https://github.com/JaidedAI/EasyOCR
[14]
Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, 2023. Mammut: A simple architecture for joint learning for multimodal tasks. arXiv preprint arXiv:2303.16839 (2023).
[15]
Richard E Ladner, Melody Y Ivory, Rajesh Rao, Sheryl Burgstahler, Dan Comden, Sangyun Hahn, Matthew Renzelmann, Satria Krisnandi, Mahalakshmi Ramasamy, Beverly Slabosky, 2005. Automating tactile graphics translation. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility. 150–157.
[16]
Beom-Chan Lee, Junhun Lee, Jongeun Cha, Changhoon Seo, and Jeha Ryu. 2005. Immersive live sports experience with vibrotactile sensation. In Human-Computer Interaction-INTERACT 2005: IFIP TC13 International Conference, Rome, Italy, September 12-16, 2005. Proceedings 10. Springer, 1042–1045.
[17]
Professional Educator Licensing and Standards Board (PELSB). [n. d.]. Descriptive Videos. https://education.mn.gov/MDE/fam/mbtbl/descr/ 2023.
[18]
Jongho Lim, Yongjae Yoo, Hanseul Cho, and Seungmoon Choi. 2019. TouchPhoto: Enabling Independent Picture Taking and Understanding for Visually-Impaired Users(ICMI ’19). Association for Computing Machinery, New York, NY, USA, 124–134. https://doi.org/10.1145/3340555.3353728
[19]
Jongho Lim, Yongjae Yoo, and Seungmoon Choi. 2019. Guidance-based two-dimensional haptic contour rendering for accessible photography. In 2019 IEEE World Haptics Conference (WHC). IEEE, 401–406.
[20]
Emma Murphy, Enda Bates, and Dónal Fitzpatrick. 2010. Designing auditory cues to enhance spoken mathematics for visually impaired users. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. 75–82.
[21]
Braille Authority of North America. 2010. The Braille Authority of North America. Guidelines and Standards for Tactile Graphics. https://www.brailleauthority.org/sites/default/files/tg/web-manual/index.html
[22]
Hiroyuki Ohshima, Makoto Kobayashi, and Shigenobu Shimada. 2021. Development of blind football play-by-play system for visually impaired spectators: tangible sports. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–6.
[23]
Peter Parente. 2003. Audio enriched links: web page previews for blind users. In Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility. 2–8.
[24]
J. Regimbal, J. R. Blum, C. Kuo, and J. R. Cooperstock. 2024. IMAGE: An Open-Source, Extensible Framework for Deploying Accessible Audio and Haptic Renderings of Web Graphics. ACM Transactions on Accessible Computing (2024). https://doi.org/doi.org/10.1145/3665223
[25]
Andrii Soviak, Vikas Ashok, Yevgen Borodin, Yury Puzis, and IV Ramakrishnan. 2015. Feel the Web: Towards the Design of Haptic Screen Interfaces for Accessible Web Browsing. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. 391–392.
[26]
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105–6114.
[27]
Kenta Tanabe, Akifumi Takahashi, Keisuke Hoshino, Daichi Ogawa, Taku Hachisu, and Hiroyuki Kajimoto. 2018. HapTONE: haptic instrument for enriched musical play (II)—system detail. In Haptic Interaction: Science, Engineering and Design 2. Springer, 461–465.
[28]
Haruya Uematsu, Daichi Ogawa, Ryuta Okazaki, Taku Hachisu, and Hiroyuki Kajimoto. 2016. HALUX: projection-based interactive skin for digital sports. In ACM SIGGRAPH 2016 Emerging Technologies. 1–2.
[29]
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, 2023. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14408–14419.
[30]
Xiyue Wang, Seita Kayukawa, Hironobu Takagi, and Chieko Asakawa. 2022. Bentomuseum: 3d and layered interactive museum map for blind visitors. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–14.
[31]
Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, 2023. mplug-2: A modularized multi-modal foundation model across text, image and video. In International Conference on Machine Learning. PMLR, 38728–38748.
[32]
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10714–10726.
[33]
Beste F Yuksel, Pooyan Fazli, Umang Mathur, Vaishali Bisht, Soo Jung Kim, Joshua Junhee Lee, Seung Jung Jin, Yue-Ting Siu, Joshua A Miele, and Ilmi Yoon. 2020. Human-in-the-loop machine learning to increase video accessibility for visually impaired and blind users. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 47–60.
Index Terms
- Automatic Video-to-Audiotactile Conversion of Golf Broadcasting on A Refreshable Pin Array
Recommendations
Refreshable Braille oral appliance
W4A '15: Proceedings of the 12th International Web for All ConferenceThe "Refreshable Braille Oral Appliance" (or RBOA) is a refreshable display embedded in an oral retainer connected to a wireless microphone and through a smart phone application.
Previous research was conducted as an investigation to determine if ...
Accessible smartphones for blind users: A case study for a wayfinding system
While progress on assistive technologies have been made, some blind users still face several problems opening and using basic functionalities when interacting with touch interfaces. Sometimes, people with visual impairments may also have problems ...
VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and TechnologyThe world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks to thermostats and checkout terminals. Blind people cannot independently use such devices without at least first learning their layout,...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
396 pages
ISBN:9798400710889
DOI:10.1145/3677386
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 86 of 279 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Oct 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in