ChitChatGuide: Conversational Interaction Using Large Language Models for Assisting People with Visual Impairments to Explore a Shopping Mall
Authors: 
Yuka Kaniwa, Masaki Kuribayashi, 
Seita Kayukawa, 
Daisuke Sato, 
Hironobu Takagi, Chieko Asakawa, Shigeo MorishimaAuthors Info & Claims
Article No.: 247, Pages 1 - 25
Abstract
To enable people with visual impairments (PVI) to explore shopping malls, it is important to provide information for selecting destinations and obtaining information based on the individual's interests. We achieved this through conversational interaction by integrating a large language model (LLM) with a navigation system. ChitChatGuide allows users to plan a tour through contextual conversations, receive personalized descriptions of surroundings based on transit time, and make inquiries during navigation. We conducted a study in a shopping mall with 11 PVI, and the results reveal that the system allowed them to explore the facility with increased enjoyment. The LLM-based conversational interaction, by understanding vague and context-based questions, enabled the participants to explore unfamiliar environments effectively. The personalized and in-situ information generated by the LLM was both useful and enjoyable. Considering the limitations we identified, we discuss the criteria for integrating LLMs into navigation systems to enhance the exploration experiences of PVI.
Supplemental Material
ZIP File
Supplemental video
- Download
- 187.58 MB
ZIP File
This folder has two files: "README.txt" and "ChitChatGuide_CameraReady_Appendix.pdf." The "README.txt" file explains how the folder is organized. The "ChitChatGuide_CameraReady_Appendix.pdf" file is the appendix file. The "ChitChatGuide_Cameraready_Appendix.pdf" file has examples of prompts and responses from GPT-4 model, a conversation example in tour planning, and examples of generated POI descriptions.
- Download
- 297.69 KB
References
[1]
Ali Abdolrahmani, Maya Howes Gupta, Mei-Lian Vader, Ravi Kuber, and Stacy Branham. 2021. Towards More Transactional Voice Assistants: Investigating the Potential for a Multimodal Voice-Activated Indoor Navigation Assistant for Blind and Sighted Travelers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 495, 16 pages. https://doi.org/10.1145/3411764.3445638
[2]
Ali Abdolrahmani, Ravi Kuber, and Amy Hurst. 2016. An empirical investigation of the situationally-induced impairments experienced by blind mobile device users. In Proceedings of the 13th International Web for All Conference. ACM, New York, NY, USA, 1--8. https://doi.org/10.1145/2899475.2899482
[3]
ACCESSIBLEJAPAN. 2024. Nihonbashi, Inclusion and Technology. Retrieved in February, 2024 from https://www.accessible-japan.com/nihonbashi-inclusion-and-technology/.
[4]
Dragan Ahmetovic, Cole Gleason, Chengxiong Ruan, Kris Kitani, Hironobu Takagi, and Chieko Asakawa. 2016. NavCog: a navigational cognitive assistant for the blind. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, New York, NY, USA, 90--99. https://doi.org/10.1145/2935334.2935361
[5]
Saleh Alghamdi, Ron van Schyndel, and Ahmed Alahmadi. 2013. Indoor navigational aid using active RFID and QR-code for sighted and blind people. In 2013 IEEE Eighth International Conference on Intelligent Sensors, Sensor Networks and Information Processing. IEEE, Piscataway, NJ, USA, 18--22. https://doi.org/10.1109/ISSNIP.2013.6529756
[6]
Abdulrhman Alkhanifer and Stephanie Ludi. 2015. Disorientation Factors that Affect the Situation Awareness of the Visually Impaired Individuals in Unfamiliar Indoor Environments. In Universal Access in Human-Computer Interaction. Access to the Human Environment and Culture. Springer International Publishing, Cham, 89--100. https://doi.org/10.1007/978--3--319--20687--5_9
[7]
Saki Asakawa, Jo ao Guerreiro, Dragan Ahmetovic, Kris M Kitani, and Chieko Asakawa. 2018. The present and future of museum accessibility for people with visual impairments. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 382--384. https://doi.org/10.1145/3234695.3240997
[8]
Watson Assistant. 2024. IBM Watson. Retrieved in February, 2024 from https://www.ibm.com/watson.
[9]
Amos Azaria, Rina Azoulay, and Shulamit Reches. 2023. ChatGPT is a Remarkable Tool -- For Experts. arxiv: 2306.03102 [cs.HC]
[10]
Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, Carl Yang, Yue Cheng, and Liang Zhao. 2024. Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models. arxiv: 2401.00625 [cs.LG]
[11]
Jan Balata, Zdenek Mikovec, and Pavel Slavik. 2018. Conversational Agents for Physical World Navigation. Springer International Publishing, Cham, 61--83. https://doi.org/10.1007/978--3--319--95579--7_4
[12]
Nikola Banovic, Rachel L. Franz, Khai N. Truong, Jennifer Mankoff, and Anind K. Dey. 2013. Uncovering information needs for independent spatial learning for users who are visually impaired. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, Article 24, 8 pages. https://doi.org/10.1145/2513383.2513445
[13]
Jakub Berka, Jan Balata, Catholijn M Jonker, Zdenek Mikovec, M Birna van Riemsdijk, and Myrthe L Tielman. 2022. Misalignment in semantic user model elicitation via conversational agents: a case study in navigation support for visually impaired people. International Journal of Human-Computer Interaction, Vol. 38, 18--20 (2022), 1909--1925. https://doi.org/10.1080/10447318.2022.2059925
[14]
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 333--342. https://doi.org/10.1145/1866029.1866080
[15]
Jeffrey R Blum, Mathieu Bouchard, and Jeremy R Cooperstock. 2011. What's around me? Spatialized audio augmented reality for blind users with a smartphone. In International Conference on Mobile and Ubiquitous Systems: Computing, Networking, and Services. Springer Berlin Heidelberg, Berlin, Heidelberg, 49--62. https://doi.org/10.1007/978--3--642--30973--1_5
[16]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 1877--1901.
[17]
Kristina Bäckström. 2006. Understanding recreational shopping: A new approach. The International Review of Retail, Distribution and Consumer Research, Vol. 16, 2 (2006), 143--158. https://doi.org/10.1080/09593960600572167
[18]
Hsuan-Eng Chen, Yi-Ying Lin, Chien-Hsing Chen, and I-Fang Wang. 2015. BlindNavi: A navigation app for the visually impaired smartphone user. In Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems. ACM, New York, NY, USA, 19--24. https://doi.org/10.1145/2702613.2726953
[19]
Elizabeth R Chrastil, Katherine R Sherrill, Michael E Hasselmo, and Chantal E Stern. 2015. There and back again: hippocampus and retrosplenial cortex track homing distance during human path integration. Journal of Neuroscience, Vol. 35, 46 (2015), 15442--15452. https://doi.org/10.1523/JNEUROSCI.1209--15.2015
[20]
Elizabeth R Chrastil and William H Warren. 2012. Active and passive contributions to spatial learning. Psychonomic bulletin & review, Vol. 19 (2012), 1--23. https://doi.org/10.3758/s13423-011-0182-x
[21]
Elizabeth R Chrastil and William H Warren. 2013. Active and passive spatial learning in human navigation: acquisition of survey knowledge. Journal of experimental psychology: learning, memory, and cognition, Vol. 39, 5 (2013), 1520--1537. https://doi.org/10.1037/a0032382
[22]
Elizabeth R Chrastil and William H Warren. 2015. Active and passive spatial learning in human navigation: acquisition of graph knowledge. Journal of experimental psychology: learning, memory, and cognition, Vol. 41, 4 (2015), 1162. https://doi.org/10.1037/xlm0000082
[23]
Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, Vincent Wade, and Benjamin R. Cowan. 2019. What makes a good conversation? Challenges in designing truly conversational agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 475, 12 pages. https://doi.org/10.1145/3290605.3300705
[24]
D. D. Clark-Carter, A. D. Heyes, and C. I. Howarth. 1986. The efficiency and walking speed of visually impaired people. Ergonomics, Vol. 29, 6 (1986), 779--789. https://doi.org/10.1080/00140138608968314
[25]
Gregory D. Clemenson, Caden M. Henningfield, and Craig E. L. Stark. 2019. Improving hippocampal memory through the experience of a rich Minecraft environment. Frontiers in Behavioral Neuroscience, Vol. 13, Article 57 (2019), 13 pages. https://doi.org/10.3389/fnbeh.2019.00057
[26]
Sanorita Dey, Karrie Karahalios, and Wai-Tat Fu. 2018. Getting there and beyond: Incidental learning of spatial knowledge with turn-by-turn directions and location updates in navigation interfaces. In Proceedings of the 2018 ACM Symposium on Spatial User Interaction. ACM, New York, NY, USA, 100--110. https://doi.org/10.1145/3267782.3267783
[27]
Joel Eapen and Adhithyan V S. 2023. Personalization and Customization of LLM Responses. International Journal of Research Publication and Reviews, Vol. 4, 12 (2023), 2617--2627. https://doi.org/10.55248/gengpi.4.1223.123512
[28]
Christin Engel, Karin Müller, Angela Constantinescu, Claudia Loitsch, Vanessa Petrausch, Gerhard Weber, and Rainer Stiefelhagen. 2020. Travelling More Independently: A Requirements Analysis for Accessible Journeys to Unknown Buildings for People with Visual Impairments. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, Article 27, 11 pages. https://doi.org/10.1145/3373625.3417022
[29]
Navid Fallah, Ilias Apostolopoulos, Kostas Bekris, and Eelke Folmer. 2012. The user as a sensor: navigating users with visual impairments in indoor spaces using tactile landmarks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 425--432. https://doi.org/10.1145/2207676.2207735
[30]
Mohammadreza Farrokhnia, Seyyed Kazem Banihashem, Omid Noroozi, and Arjen Wals. 2024. A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, Vol. 61, 3 (2024), 460--474. https://doi.org/10.1080/14703297.2023.2195846
[31]
Bhanuka Gamage, Thanh-Toan Do, Nicholas Seow Chiang Price, Arthur Lowery, and Kim Marriott. 2023. What do Blind and Low-Vision People Really Want from Assistive Smart Devices? Comparison of the Literature with a Focus Study. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, Article 30, 21 pages. https://doi.org/10.1145/3597638.3608955
[32]
Nicholas A Giudice. 2018. Navigating without vision: Principles of blind spatial cognition. In Handbook of behavioral and cognitive geography. Edward Elgar Publishing, United Kingdom, 260--288. https://doi.org/10.4337/9781784717544.00024
[33]
Cole Gleason, Alexander J. Fiannaca, Melanie Kneisel, Edward Cutrell, and Meredith Ringel Morris. 2018. FootNotes: Geo-referenced Audio Annotations for Nonvisual Exploration. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 2, 3, Article 109 (2018), 24 pages. https://doi.org/10.1145/3264919
[34]
Jo ao Guerreiro, Daisuke Sato, Saki Asakawa, Huixu Dong, Kris M Kitani, and Chieko Asakawa. 2019. CaBot: Designing and Evaluating an Autonomous Navigation Robot for Blind People. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 68--82. https://doi.org/10.1145/3308561.3353771
[35]
Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual Programming: Compositional Visual Reasoning Without Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, USA, 14953--14962. https://doi.org/10.1109/CVPR52729.2023.01436
[36]
HULOP. 2018. Human-scale Localization Platform (HULOP). Retrieved in May, 2024 from https://github.com/hulop.
[37]
Dhruv Jain. 2014. Path-guided indoor navigation for the visually impaired using minimal building retrofitting. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 225--232. https://doi.org/10.1145/2661334.2661359
[38]
Gaurav Jain, Yuanyang Teng, Dong Heon Cho, Yunhao Xing, Maryam Aziz, and Brian A Smith. 2023. "I Want to Figure Things Out": Supporting Exploration in Navigation for People with Visual Impairments. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW1, Article 63 (2023), 28 pages. https://doi.org/10.1145/3579496
[39]
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys, Vol. 55, 12, Article 248 (2023), 38 pages. https://doi.org/10.1145/3571730
[40]
Vaishnav Kameswaran, Alexander J. Fiannaca, Melanie Kneisel, Amy Karlson, Edward Cutrell, and Meredith Ringel Morris. 2020. Understanding In-Situ Use of Commonly Available Navigation Technologies by People with Visual Impairments. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, Article 28, 12 pages. https://doi.org/10.1145/3373625.3416995
[41]
Rie Kamikubo, Hernisa Kacorri, and Chieko Asakawa. 2024. "We are at the mercy of others' opinion": Supporting Blind People in Recreational Window Shopping with AI-infused Technology. arxiv: 2405.06611 [cs.HC]
[42]
Seita Kayukawa, Daisuke Sato, Masayuki Murata, Tatsuya Ishihara, Akihiro Kosugi, Hironobu Takagi, Shigeo Morishima, and Chieko Asakawa. 2022. How Users, Facility Managers, and Bystanders Perceive and Accept a Navigation Robot for Visually Impaired People in Public Buildings. In Proceedings of the 31st IEEE International Conference on Robot & Human Interactive Communication. IEEE, Piscataway, NJ, USA, 546--553. https://doi.org/10.1109/RO-MAN53752.2022.9900717
[43]
Seita Kayukawa, Daisuke Sato, Masayuki Murata, Tatsuya Ishihara, Hironobu Takagi, Shigeo Morishima, and Chieko Asakawa. 2023. Enhancing Blind Visitor's Autonomy in a Science Museum Using an Autonomous Navigation Robot. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 541, 14 pages. https://doi.org/10.1145/3544548.3581220
[44]
Sulaiman Khan, Shah Nazir, and Habib Ullah Khan. 2021. Analysis of Navigation Assistants for Blind and Visually Impaired People: A Systematic Review. IEEE Access, Vol. 9 (2021), 26712--26734. https://doi.org/10.1109/ACCESS.2021.3052415
[45]
Jee-Eun Kim, Masahiro Bessho, Shinsuke Kobayashi, Noboru Koshizuka, and Ken Sakamura. 2016. Navigating visually impaired travelers in a large train station using smartphone and bluetooth low energy. In Proceedings of the 31st Annual ACM Symposium on Applied Computing. ACM, New York, NY, USA, 604--611. https://doi.org/10.1145/2851613.2851716
[46]
Kyoko Konishi and Véronique D Bohbot. 2013. Spatial navigational strategies correlate with gray matter in the hippocampus of healthy older adults tested in a virtual maze. Frontiers in aging neuroscience, Vol. 5 (2013), 28885. https://doi.org/10.3389/fnagi.2013.00001
[47]
Bineeth Kuriakose, Raju Shrestha, and Frode Eika Sandnes. 2020. Tools and Technologies for Blind and Visually Impaired Navigation Support: A Review. IETE Technical Review, Vol. 39, 1 (2020), 3--18. https://doi.org/10.1080/02564602.2020.1819893
[48]
Masaki Kuribayashi, Seita Kayukawa, Hironobu Takagi, Chieko Asakawa, and Shigeo Morishima. 2021. LineChaser: A Smartphone-Based Navigation System for Blind People to Stand in Lines. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 33, 13 pages. https://doi.org/10.1145/3411764.3445451
[49]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages.
[50]
Bing Li, J Pablo Munoz, Xuejian Rong, Jizhong Xiao, Yingli Tian, and Aries Arditi. 2016. ISANA: wearable context-aware indoor assistive navigation with obstacle avoidance for the blind. In European Conference on Computer Vision. Springer International Publishing, Cham, 448--462. https://doi.org/10.1007/978--3--319--48881--3_31
[51]
Chen-Lung Lu, Zi-Yan Liu, Jui-Te Huang, Ching-I Huang, Bo-Hui Wang, Yi Chen, Nien-Hsin Wu, Hsueh-Cheng Wang, Laura Giarré, and Pei-Yi Kuo. 2021. Assistive Navigation Using Deep Reinforcement Learning Guiding Robot With UWB/Voice Beacons and Semantic Feedbacks for Blind and Visually Impaired People. Frontiers in Robotics and AI, Vol. 8, Article 654132 (2021), 15 pages. https://doi.org/10.3389/frobt.2021.654132
[52]
Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 5286--5297. https://doi.org/10.1145/2858036.2858288
[53]
Eleanor A Maguire, David G Gadian, Ingrid S Johnsrude, Catriona D Good, John Ashburner, Richard SJ Frackowiak, and Christopher D Frith. 2000. Navigation-related structural change in the hippocampi of taxi drivers. Proceedings of the National Academy of Sciences, Vol. 97, 8 (2000), 4398--4403. https://doi.org/10.1073/pnas.070039597
[54]
Kanak Manjari, Madhushi Verma, and Gaurav Singal. 2020. A survey on Assistive Technology for visually impaired. Internet of Things, Vol. 11, Article 100188 (2020). https://doi.org/10.1016/j.iot.2020.100188
[55]
Madalin Matei, Lenuta Alboaie, and Adrian Iftene. 2022. Safety Navigation using a Conversational User Interface For Visually Impaired People. Procedia Computer Science, Vol. 207 (2022), 1164--1173. https://doi.org/10.1016/j.procs.2022.09.172
[56]
Karin Müller, Christin Engel, Claudia Loitsch, Rainer Stiefelhagen, and Gerhard Weber. 2022. Traveling More Independently: A Study on the Diverse Needs and Challenges of People with Visual or Mobility Impairments in Unfamiliar Indoor Environments. ACM Transactions on Accessible Computing, Vol. 15, 2, Article 13 (2022), 44 pages. https://doi.org/10.1145/3514255
[57]
Masayuki Murata, Dragan Ahmetovic, Daisuke Sato, Hironobu Takagi, Kris M. Kitani, and Chieko Asakawa. 2018. Smartphone-based Indoor Localization for Blind Navigation across Building Complexes. In 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, Piscataway, NJ, USA, 1--10. https://doi.org/10.1109/PERCOM.2018.8444593
[58]
OpenAI. 2024. Gpt-4. Retrieved in February, 2024 from https://openai.com/research/gpt-4.
[59]
James Jie Pan, Jianguo Wang, and Guoliang Li. 2023. Survey of Vector Database Management Systems. arxiv: 2310.14021 [cs.DB]
[60]
Suraj R. Pardeshi, Vikul J. Pawar, Kailas D. Kharat, and Sachin Chavan. 2021. Assistive Technologies for Visually Impaired Persons Using Image Processing Techniques -- A Survey. In International Conference on Recent Trends in Image Processing and Pattern Recognition. Springer Singapore, Singapore, 95--110. https://doi.org/10.1007/978--981--16-0507--9_9
[61]
Manoj Penmetcha, Arabinda Samantaray, and Byung-Cheol Min. 2017. Smartresponse: Emergency and non-emergency response for smartphone based indoor localization applications. In International Conference on Human-Computer Interaction. Springer International Publishing, Cham, 398--404. https://doi.org/10.1007/978--3--319--58753-0_57
[62]
Emanuele Pucci, Isabella Possaghi, Claudia Maria Cutrupi, Marcos Baez, Cinzia Cappiello, and Maristella Matera. 2023. Defining Patterns for a Conversational Web. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 118, 17 pages. https://doi.org/10.1145/3544548.3581145
[63]
Aubrey Ramatla and Anne Mastamet-Mason. 2013. The decision-making processes of visually impaired consumers in an apparel retail environment. In 2013 DEFSA conference Design Cultures: Encultured Design. Design Education Forum of South Africa, 220--228.
[64]
Lisa Ran, Sumi Helal, and Steve Moore. 2004. Drishti: an integrated indoor/outdoor blind navigation system and service. In Second IEEE Annual Conference on Pervasive Computing and Communications. IEEE, Piscataway, NJ, USA, 23--30. https://doi.org/10.1109/PERCOM.2004.1276842
[65]
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Singapore, 431--445. https://doi.org/10.18653/v1/2023.emnlp-demo.40
[66]
Microsoft Research. 2023. Microsoft Soundscape. Retrieved in February, 2024 from https://www.microsoft.com/en-us/research/product/soundscape/.
[67]
Timothy H Riehle, P Lichter, and Nicholas A Giudice. 2008. An indoor navigation system to support the visually impaired. In 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, Piscataway, NJ, USA, 4435--4438. https://doi.org/10.1109/IEMBS.2008.4650195
[68]
Bishal Santra, Sakya Basak, Abhinandan De, Manish Gupta, and Pawan Goyal. 2023. Frugal Prompting for Dialog Models. arxiv: 2305.14919 [cs.CL]
[69]
Daisuke Sato, Uran Oh, Jo ao Guerreiro, Dragan Ahmetovic, Kakuya Naito, Hironobu Takagi, Kris M Kitani, and Chieko Asakawa. 2019. NavCog3 in the wild: Large-scale blind indoor navigation assistant with semantic features. ACM Transactions on Accessible Computing, Vol. 12, 3, Article 14 (2019), 30 pages. https://doi.org/10.1145/3340319
[70]
Ben Shneiderman. 1996. The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In Proceedings of the 1996 IEEE Symposium on Visual Languages. IEEE Computer Society, USA, 336--343. https://doi.org/10.1109/VL.1996.545307
[71]
Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text Classification via Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Sentosa, Singapore, 8990--9005. https://doi.org/10.18653/v1/2023.findings-emnlp.603
[72]
Zhewei Sun, Qian Hu, Rahul Gupta, Richard Zemel, and Yang Xu. 2024. Toward Informal Language Processing: Knowledge of Slang in Large Language Models. arxiv: 2404.02323 [cs.CL]
[73]
Sandra Tullio-Pow, Hong Yu, and Megan Strickfaden. 2021. Do You See What I See? The shopping experiences of people with visual impairment. Interdisciplinary Journal of Signage and Wayfinding, Vol. 5, 1 (2021), 42--61. https://doi.org/10.15763/issn.2470--9670.2021.v5.i1.a69
[74]
Beatrice Vincenzi, Alex S. Taylor, and Simone Stumpf. 2021. Interdependence in Action: People with Visual Impairments and their Guides Co-constituting Common Spaces. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1, Article 69 (2021), 33 pages. https://doi.org/10.1145/3449143
[75]
Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction with Mobile UI Using Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 432, 17 pages. https://doi.org/10.1145/3544548.3580895
[76]
Luyao Wang, Qihe Chen, Yan Zhang, Ziang Li, Tingmin Yan, Fan Wang, Guyue Zhou, and Jiangtao Gong. 2023. Can Quadruped Navigation Robots be Used as Guide Dogs?arxiv: 2210.08727 [cs.HC]
[77]
Liyang Wang, Jinxin Zhao, and Liangjun Zhang. 2021. Navdog: robotic navigation guide dog via model predictive control and human-robot modeling. In Proceedings of the 36th Annual ACM Symposium on Applied Computing. ACM, New York, NY, USA, 815--818. https://doi.org/10.1145/3412841.3442098
[78]
Lan Xia. 2010. An examination of consumer browsing behaviors. Qualitative Market Research: An International Journal, Vol. 13, 2 (2010), 154--173. https://doi.org/10.1108/13522751011032593
[79]
Chris Yoon, Ryan Louie, Jeremy Ryan, MinhKhang Vu, Hyegi Bang, William Derksen, and Paul Ruvolo. 2019. Leveraging Augmented Reality to Create Apps for People with Visual Disabilities: A Case Study in Indoor Navigation. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility. ACM, New York, NY, USA, 210--221. https://doi.org/10.1145/3308561.3353788
[80]
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, Vol. 12 (2024), 39--57. https://doi.org/10.1162/tacl_a_00632
[81]
Yan Zhang, Ziang Li, Haole Guo, Luyao Wang, Qihe Chen, Wenjie Jiang, Mingming Fan, Guyue Zhou, and Jiangtao Gong. 2023. "I am the follower, also the boss": Exploring Different Levels of Autonomy and Machine Forms of Guiding Robots for the Visually Impaired. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 542, 22 pages. https://doi.org/10.1145/3544548.3580884
[82]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A Survey of Large Language Models. arxiv: 2303.18223 [cs.CL]
[83]
Yuhang Zhao, Elizabeth Kupferstein, Hathaitorn Rojnirun, Leah Findlater, and Shiri Azenkot. 2020. The Effectiveness of Visual and Audio Wayfinding Guidance on Smartglasses for People with Low Vision. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376516
Index Terms
- ChitChatGuide: Conversational Interaction Using Large Language Models for Assisting People with Visual Impairments to Explore a Shopping Mall
Recommendations
Navigating Real-World Challenges: A Quadruped Robot Guiding System for Visually Impaired People in Diverse Environments
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing SystemsBlind and Visually Impaired (BVI) people find challenges in navigating unfamiliar environments, even using assistive tools such as white canes or smart devices. Increasingly affordable quadruped robots offer us opportunities to design autonomous guides ...
PathFinder: Designing a Map-less Navigation System for Blind People in Unfamiliar Buildings
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsIndoor navigation systems with prebuilt maps have shown great potential in navigating blind people even in unfamiliar buildings. However, blind people cannot always benefit from them in every building, as prebuilt maps are expensive to build. This paper ...
Airport Accessibility and Navigation Assistance for People with Visual Impairments
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsPeople with visual impairments often have to rely on the assistance of sighted guides in airports, which prevents them from having an independent travel experience. In order to learn about their perspectives on current airport accessibility, we ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2024
1136 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 September 2024
Published in PACMHCI Volume 8, Issue MHCI
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- JSPS KAKENHI
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 242Total Downloads
- Downloads (Last 12 months)242
- Downloads (Last 6 weeks)37
Reflects downloads up to 12 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in