CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision
Authors: 
Jaewook Lee, Andrew D. Tjahjadi, Jiho Kim, Junpu Yu, Minji Park, Jiawen Zhang, 
Jon E. Froehlich, Yapeng Tian, Yuhang ZhaoAuthors Info & Claims
Article No.: 141, Pages 1 - 16
Abstract
Cooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.
Supplemental Material
MP4 File
Video Figure
- Download
- 123.79 MB
References
[1]
Shiri Azenkot and Yuhang Zhao. 2017. Designing smartglasses applications for people with low vision. SIGACCESS Access. Comput.119 (nov 2017), 19–24. https://doi.org/10.1145/3167902.3167905
[2]
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. 2023. Affordances from Human Videos as a Versatile Representation for Robotics. CVPR.
[3]
Marie Claire Bilyk, Jessica M Sontrop, Gwen E Chapman, Susan I Barr, and Linda Mamer. 2009. Food experiences and eating patterns of visually impaired and blind people. Canadian Journal of Dietetic practice and research 70, 1 (2009), 13–18.
[4]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
[5]
Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597. https://doi.org/10.1080/2159676X.2019.1628806
[6]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. 2019. MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv preprint arXiv:1906.07155 (2019).
[7]
Fu-Jen Chu, Ruinian Xu, Landan Seguin, and Patricio A. Vela. 2019. Toward Affordance Detection and Ranking on Novel Objects for Real-World Robotic Manipulation. IEEE Robotics and Automation Letters 4, 4 (2019), 4070–4077. https://doi.org/10.1109/LRA.2019.2930364
[8]
Fu-Jen Chu, Ruinian Xu, and Patricio A. Vela. 2019. Learning Affordance Segmentation for Real-World Robotic Manipulation via Synthetic Images. IEEE Robotics and Automation Letters 4, 2 (2019), 1140–1147. https://doi.org/10.1109/LRA.2019.2894439
[9]
Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. 2018. Learning to Act Properly: Predicting and Explaining Affordances from Images. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 975–983. https://doi.org/10.1109/CVPR.2018.00108
[10]
Anne Lesley Corn and Jane N Erin. 2010. Foundations of low vision: Clinical and functional perspectives. American Foundation for the Blind.
[11]
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In Proceedings of the European Conference on Computer Vision (ECCV).
[12]
Ashley D Deemer, Christopher K Bradley, Nicole C Ross, Danielle M Natale, Rath Itthipanichpong, Frank S Werblin, and Robert W Massof. 2018. Low vision enhancement with head-mounted video display systems: are we there yet?Optometry and Vision Science 95, 9 (2018), 694–703.
[13]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
[14]
MR Everingham, BT Thomas, T Troscianko, 1999. Head-mounted mobility aid for low vision using scene classification techniques. The International Journal of Virtual Reality 3, 4 (1999), 3.
[15]
Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and Angela Carey Lin. 2019. Deaf and Hard-of-hearing Individuals’ Preferences for Wearable and Mobile Sound Awareness Technologies. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300276
[16]
Dylan R. Fox, Ahmad Ahmadzada, Clara Tenia Wang, Shiri Azenkot, Marlena A. Chu, Roberto Manduchi, and Emily A. Cooper. 2023. Using augmented reality to cue obstacles for people with low vision. Opt. Express 31, 4 (Feb 2023), 6827–6848. https://doi.org/10.1364/OE.479258
[17]
Patricia Garrido-Vásquez and Anna Schubö. 2014. Modulation of visual attention by object affordance. Frontiers in Psychology 5 (2014), 70664.
[18]
James Jerome Gibson. 1966. The senses considered as perceptual systems. (1966).
[19]
James J Gibson. 1977. The theory of affordances. Hilldale, USA 1, 2 (1977), 67–82.
[20]
James J Gibson. 2014. The ecological approach to visual perception: classic edition. Psychology press.
[21]
Anshul Gupta, Juraj Mesik, Stephen A Engel, Rebecca Smith, Mark Schatza, Aurélie Calabrese, Frederik J Van Kuijk, Arthur G Erdman, and Gordon E Legge. 2018. Beneficial effects of spatial remapping for reading with simulated central field loss. Investigative ophthalmology & visual science 59, 2 (2018), 1105–1112.
[22]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961–2969.
[23]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
[24]
Stephen L Hicks, Iain Wilson, Louwai Muhammed, John Worsfold, Susan M Downes, and Christopher Kennard. 2013. A depth-based head-mounted visual display to aid navigation in partially sighted individuals. PloS one 8, 7 (2013), e67695.
[25]
Jonathan Huang, Max Kinateder, Matt J Dunn, Wojciech Jarosz, Xing-Dong Yang, and Emily A Cooper. 2019. An augmented reality sign-reading assistant for users with reduced vision. PloS one 14, 1 (2019), e0210630.
[26]
Alex D Hwang and Eli Peli. 2014. An augmented-reality edge enhancement application for Google Glass. Optometry and vision science 91, 8 (2014), 1021–1030.
[27]
Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich. 2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (Hong Kong, China) (DIS ’18 Companion). Association for Computing Machinery, New York, NY, USA, 7–11. https://doi.org/10.1145/3197391.3205404
[28]
Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, and Jon E. Froehlich. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 241–250. https://doi.org/10.1145/2702123.2702393
[29]
Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, and Jon Froehlich. 2018. Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (Galway, Ireland) (ASSETS ’18). Association for Computing Machinery, New York, NY, USA, 81–92. https://doi.org/10.1145/3234695.3236362
[30]
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. Ultralytics YOLO. https://github.com/ultralytics/ultralytics
[31]
Nabila Jones, Hannah Elizabeth Bartlett, and Richard Cooke. 2019. An analysis of the impact of visual impairment on activities of daily living and vision-related quality of life in a visually impaired adult population. British Journal of Visual Impairment 37, 1 (2019), 50–63.
[32]
Avyay Ravi Kashyap. 2020. Behaviors, Problems and strategies of visually impaired persons during meal preparation in the Indian context: challenges and opportunities for Design. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1–3.
[33]
Minyung Kim, Sooyoung Hwang, Kyoungmin Choi, Youkeun Oh, and Dokshin Lim. 2022. Vision-Based Cooking Assistance System for Visually Impaired People. In International Conference on Human-Computer Interaction. Springer, 540–547.
[34]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023).
[35]
MiYoung Kwon, Chaithanya Ramachandra, PremNandhini Satgunam, Bartlett W Mel, Eli Peli, and Bosco S Tjan. 2012. Contour enhancement benefits older adults with simulated central field loss. Optometry and vision science 89, 9 (2012), 1374–1384.
[36]
Florian Lang and Tonja Machulla. 2021. Pressing a Button You Cannot See: Evaluating Visual Designs to Assist Persons with Low Vision through Augmented Reality. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (Osaka, Japan) (VRST ’21). Association for Computing Machinery, New York, NY, USA, Article 39, 10 pages. https://doi.org/10.1145/3489849.3489873
[37]
Jaewook Lee, Devesh P. Sarda, Eujean Lee, Amy Lee, Jun Wang, Adrian Rodriguez, and Jon E. Froehlich. 2023. Towards Real-time Computer Vision and Augmented Reality to Support Low Vision Sports: A Demonstration of ARTennis. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (, San Francisco, CA, USA,) (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 81, 3 pages. https://doi.org/10.1145/3586182.3615815
[38]
Franklin Mingzhe Li, Jamie Dorst, Peter Cederberg, and Patrick Carrington. 2021. Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (, Virtual Event, USA,) (ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article 30, 11 pages. https://doi.org/10.1145/3441852.3471215
[39]
Franklin Mingzhe Li, Michael Xieyang Liu, Shaun K Kane, and Patrick Carrington. 2024. A Contextual Inquiry of People with Vision Impairments in Cooking. arXiv preprint arXiv:2402.15108 (2024).
[40]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755.
[41]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022.
[42]
David S Loshin and Richard D Juday. 1989. The programmable remapper: clinical applications for patients with field defects. Optometry and Vision Science 66, 6 (1989), 389–395.
[43]
Timo Luddecke and Florentin Worgotter. 2017. Learning to segment affordances. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 769–776.
[44]
Gang Luo and Eli Peli. 2006. Use of an augmented-vision device for visual search by patients with tunnel vision. Investigative ophthalmology & visual science 47, 9 (2006), 4152–4159.
[45]
Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. 2022. Learning Affordance Grounding from Exocentric Images. In CVPR.
[46]
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen. 2022. RTMDet: An Empirical Study of Designing Real-Time Object Detectors. arxiv:2212.07784 [cs.CV]
[47]
Meethu Malu and Leah Findlater. 2014. "OK Glass?" A Preliminary Exploration of Google Glass for Persons with Upper Body Motor Impairments. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (Rochester, New York, USA) (ASSETS ’14). Association for Computing Machinery, New York, NY, USA, 267–268. https://doi.org/10.1145/2661334.2661400
[48]
Meethu Malu and Leah Findlater. 2015. Personalized, Wearable Control of a Head-mounted Display for Users with Upper Body Motor Impairments. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 221–230. https://doi.org/10.1145/2702123.2702188
[49]
Robert W Massof and Douglas L Rickman. 1992. Obstacles encountered in the development of the low vision enhancement system. Optometry and vision science 69, 1 (1992), 32–41.
[50]
Robert W Massof, Douglas L Rickman, Peter A Lalle, 1994. Low vision enhancement system. Johns Hopkins APL Technical Digest 15, 2 (1994), 120–125.
[51]
Roisin McNaney, John Vines, Daniel Roggen, Madeline Balaam, Pengfei Zhang, Ivan Poliakov, and Patrick Olivier. 2014. Exploring the acceptability of google glass as an everyday assistive device for people with parkinson’s. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2551–2554. https://doi.org/10.1145/2556288.2557092
[52]
Ashley Miller, Joan Malasig, Brenda Castro, Vicki L. Hanson, Hugo Nicolau, and Alessandra Brandão. 2017. The Use of Smart Glasses for Lecture Comprehension by Deaf and Hard of Hearing Students. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (, Denver, Colorado, USA,) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 1909–1915. https://doi.org/10.1145/3027063.3053117
[53]
Susanna Mills, Martin White, Heather Brown, Wendy Wrieden, Dominika Kwasnicka, Joel Halligan, Shannon Robalino, and Jean Adams. 2017. Health and social determinants and outcomes of home cooking: A systematic review of observational studies. Appetite 111 (2017), 116–134.
[54]
Kenzaburo Miyawaki, Mutsuo Sano, Syunichi Yonemura, and Michiko Ode. 2012. Cooking rehabilitation support for self-reliance of cognitive dysfunction patients. In Proceedings of the ACM multimedia 2012 workshop on Multimedia for cooking and eating activities. 19–24.
[55]
Pilar Montero. 2005. Nutritional assessment and diet quality of visually impaired Spanish children. Annals of Human Biology 32, 4 (2005), 498–512.
[56]
Elizabeth D Mynatt and Wendy A Rogers. 2001. Developing technology to support the functional independence of older adults. Ageing International 27, 1 (2001), 24–41.
[57]
Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, and Nikos G. Tsagarakis. 2016. Detecting object affordances with Convolutional Neural Networks. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2765–2770. https://doi.org/10.1109/IROS.2016.7759429
[58]
Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, and Nikos G. Tsagarakis. 2017. Object-based affordances detection with Convolutional Neural Networks and dense Conditional Random Fields. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 5908–5915. https://doi.org/10.1109/IROS.2017.8206484
[59]
Alex Olwal, Kevin Balke, Dmitrii Votintcev, Thad Starner, Paula Conn, Bonnie Chinh, and Benoit Corda. 2020. Wearable Subtitles: Augmenting Spoken Communication with Lightweight Eyewear for All-day Captioning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 1108–1120. https://doi.org/10.1145/3379337.3415817
[60]
Eli Peli. 2001. Vision multiplexing: an engineering approach to vision rehabilitation device development. Optometry and Vision Science 78, 5 (2001), 304–315.
[61]
Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (, Montreal QC, Canada,) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173867
[62]
Kyle Rector, Lauren Milne, Richard E Ladner, Batya Friedman, and Julie A Kientz. 2015. Exploring the opportunities and challenges with exercise technologies for people who are blind or low-vision. In Proceedings of the 17th international ACM SIGACCESS conference on computers & accessibility. 203–214.
[63]
Gwendolyn Rehrig, Madison Barker, Candace E Peacock, Taylor R Hayes, John M Henderson, and Fernanda Ferreira. 2022. Look at what I can do: Object affordances guide visual attention while speakers describe potential actions. Attention, Perception, & Psychophysics 84, 5 (2022), 1583–1610.
[64]
Jarek Reynolds, Chandra Kanth Nagesh, and Danna Gurari. 2024. Salient object detection for images taken by people with vision impairments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 8522–8531.
[65]
Chris Schipper and Bo Brinkman. 2017. Caption Placement on an Augmented Reality Head Worn Device. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS ’17). Association for Computing Machinery, New York, NY, USA, 365–366. https://doi.org/10.1145/3132525.3134786
[66]
Lee Stearns, Victor DeSouza, Jessica Yin, Leah Findlater, and Jon E. Froehlich. 2017. Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS ’17). Association for Computing Machinery, New York, NY, USA, 361–362. https://doi.org/10.1145/3132525.3134812
[67]
Stereolabs. 2024. zed-unity. https://github.com/stereolabs/zed-unity.
[68]
Sarit Szpiro, Yuhang Zhao, and Shiri Azenkot. 2016. Finding a store, searching for a product: a study of daily challenges of low vision people. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 61–72. https://doi.org/10.1145/2971648.2971723
[69]
Sarit Felicia Anais Szpiro, Shafeka Hashash, Yuhang Zhao, and Shiri Azenkot. 2016. How People with Low Vision Access Computing Devices: Understanding Challenges and Opportunities. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA) (ASSETS ’16). Association for Computing Machinery, New York, NY, USA, 171–180. https://doi.org/10.1145/2982142.2982168
[70]
Deanna J Taylor, Angharad E Hobby, Alison M Binns, and David P Crabb. 2016. How does age-related macular degeneration affect real-world visual ability and quality of life? A systematic review. BMJ open 6, 12 (2016), e011504.
[71]
Jan Tünnermann, Norbert Krüger, Bärbel Mertsching, and Wail Mustafa. 2015. Affordance estimation enhances artificial visual attention: Evidence from a change-blindness study. Cognitive computation 7 (2015), 526–538.
[72]
Joram J van Rheede, Iain R Wilson, Rose I Qian, Susan M Downes, Christopher Kennard, and Stephen L Hicks. 2015. Improving mobility performance in low vision with a distance-based representation of the visual scene. Investigative ophthalmology & visual science 56, 8 (2015), 4802–4809.
[73]
Ru Wang, Nihan Zhou, Tam Nguyen, Sanbrita Mondal, Bilge Mutlu, and Yuhang Zhao. 2023. Characterizing Barriers and Technology Needs in the Kitchen for Blind and Low Vision People. arXiv preprint arXiv:2310.05396 (2023).
[74]
Ru Wang, Nihan Zhou, Tam Nguyen, Sanbrita Mondal, Bilge Mutlu, and Yuhang Zhao. 2023. Practices and Barriers of Cooking Training for Blind and Low Vision People. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (, New York, NY, USA,) (ASSETS ’23). Association for Computing Machinery, New York, NY, USA, Article 57, 5 pages. https://doi.org/10.1145/3597638.3614494
[75]
Peter Washington, Catalin Voss, Aaron Kline, Nick Haber, Jena Daniels, Azar Fazel, Titas De, Carl Feinstein, Terry Winograd, and Dennis Wall. 2017. SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 112 (sep 2017), 22 pages. https://doi.org/10.1145/3130977
[76]
Kristin Williams, Karyn Moffatt, Denise McCall, and Leah Findlater. 2015. Designing Conversation Cues on a Head-Worn Display to Support Persons with Aphasia. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 231–240. https://doi.org/10.1145/2702123.2702484
[77]
Yuhang Zhao, Michele Hu, Shafeka Hashash, and Shiri Azenkot. 2017. Understanding Low Vision People’s Visual Perception on Commercial Augmented Reality Glasses. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4170–4181. https://doi.org/10.1145/3025453.3025949
[78]
Yuhang Zhao, Elizabeth Kupferstein, Brenda Veronica Castro, Steven Feiner, and Shiri Azenkot. 2019. Designing AR Visualizations to Facilitate Stair Navigation for People with Low Vision. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 387–402. https://doi.org/10.1145/3332165.3347906
[79]
Yuhang Zhao, Elizabeth Kupferstein, Hathaitorn Rojnirun, Leah Findlater, and Shiri Azenkot. 2020. The Effectiveness of Visual and Audio Wayfinding Guidance on Smartglasses for People with Low Vision. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA,) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376516
[80]
Yuhang Zhao, Elizabeth Kupferstein, Doron Tal, and Shiri Azenkot. 2018. " It Looks Beautiful but Scary" How Low Vision People Navigate Stairs and Other Surface Level Changes. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 307–320.
[81]
Yuhang Zhao, Sarit Szpiro, and Shiri Azenkot. 2015. Foresee: A customizable head-mounted vision enhancement system for people with low vision. In Proceedings of the 17th international ACM SIGACCESS conference on computers & accessibility. 239–249.
[82]
Yuhang Zhao, Sarit Szpiro, Jonathan Knighten, and Shiri Azenkot. 2016. CueSee: exploring visual cues for people with low vision to facilitate a visual search task. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 73–84. https://doi.org/10.1145/2971648.2971730
Index Terms
- CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision
Recommendations
Augmented Winter Ski with AR HMD
AH '16: Proceedings of the 7th Augmented Human International Conference 2016At time of writing, several affordable Head-Mounted Displays (HMD) are going to be released to the mass market, most of them for Virtual Reality (VR with Oculus Rift, Samsung Gear...) but also for indoor Augmented Reality (AR) with Hololens. We have ...
The First Workshop on Multiple Input Modalities and Sensations for VR/AR Interactions (MIMSVAI)
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable ComputersWith the advance of VR/AR technology, more and more VR/AR applications are emerging and have been popular among new users. Users interact with both physical or virtual worlds through multiple input modalities and sensations to create an immersive VR/AR ...
The Second Workshop on Multiple Input Modalities and Sensations for VR/AR Interactions (MIMSVAI)
UbiComp/ISWC '22 Adjunct: Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable ComputersWith the advance of VR/AR technology, more and more VR/AR applications are emerging and have been popular among new users. Interacting with virtual reality and augmented reality technologies will require the development of alternative input modalities ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
2334 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
UIST '24
UIST '24: The 37th Annual ACM Symposium on User Interface Software and Technology
October 13 - 16, 2024
PA, Pittsburgh, USA
Acceptance Rates
Overall Acceptance Rate 561 of 2,567 submissions, 22%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 753Total Downloads
- Downloads (Last 12 months)753
- Downloads (Last 6 weeks)218
Reflects downloads up to 03 Feb 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in