research-article

Public Access

BrowseWithMe: An Online Clothes Shopping Assistant for People with Visual Impairments

Authors:

Abigale J. Stangl,

Kristen Grauman,

Danna GurariAuthors Info & Claims

ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

Pages 107 - 118

https://doi.org/10.1145/3234695.3236337

Published: 08 October 2018 Publication History

Abstract

Our interviews with people who have visual impairments show clothes shopping is an important activity in their lives. Unfortunately, clothes shopping web sites remain largely inaccessible. We propose design recommendations to address online accessibility issues reported by visually impaired study participants and an implementation, which we call BrowseWithMe, to address these issues. BrowseWithMe employs artificial intelligence to automatically convert a product web page into a structured representation that enables a user to interactively ask the BrowseWithMe system what the user wants to learn about a product (e.g., What is the price? Can I see a magnified image of the pants?). This enables people to be active solicitors of the specific information they are seeking rather than passive listeners of unparsed information. Experiments demonstrate BrowseWithMe can make online clothes shopping more accessible and produce accurate image descriptions.

References

[1]

Be My Eyes. http://www.bemyeyes.org/.

[2]

http://webaim.org/projects/screenreadersurvey/.

[3]

http://www.afb.org/afbpress/pubnew.asp?DocID=aw170208.

[4]

xkcd Color Palette. https://xkcd.com/color/rgb/.

[5]

L. Von Ahn and L. Dabbish. 2004. Labeling Images With a Computer Game. In ACM Conference on Human Factors in Computing Systems (CHI). 319--326.

Digital Library

[6]

R. M. Alali. 2017. Case study of visually impaired individuals' preferences of the availability of braille clothing labels in shopping and selection of apparel. (2017).

[7]

ASOS. Online shopping for the Latest Clothes & Fashion. http://us.asos.com/. (Accessed on 09/10/2017).

[8]

M. Avila, K. Wolf, A. Brock, and N. Henze. 2016. Remote Assistance for Blind Users in Daily Life: A Survey about Be My Eyes. In International Conference on PErvasive Technologies Related to Assistive Environments (PETRA). ACM, 85.

Digital Library

[9]

J. Bigham. 2014. Making the Web Easier to See with Opportunistic Accessibility Improvement. In ACM symposium on User Interface Software and technology (UIST). 117--122.

Digital Library

[10]

J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh. 2010. VizWiz: Nearly Real-time Answers to Visual Questions. In ACM symposium on User interface software and technology (UIST). 333--342.

Digital Library

[11]

J. P. Bigham, R. S. Kaminsky, R. E. Ladner, O. M. Danielsson, and Gordon L. Hempton. 2006. Webinsight: Making Web Images Accessible. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS).

Digital Library

[12]

J. P. Bigham, I. Lin, and S. Savage. 2017. The Effects of "Not Knowing What You Don't Know" on Web Accessibility for Blind Web Users. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS).

Digital Library

[13]

E. Brady, M. R. Morris, Y. Zhong, S. White, and J. P. Bigham. 2013. Visual Challenges in the Everyday Lives of Blind People. In ACM Conference on Human Factors in Computing Systems (CHI). 2117--2126.

Digital Library

[14]

M. A. Burton, E. Brady, R. Brewer, C. Neylan, J. P. Bigham, and A. Hurst. 2012. Crowdsourcing Subjective Fashion Advice Using VizWiz: Challenges and Opportunities. In ACM SIGACCESS conference on Computers and accessibility (ASSETS). 135--142.

Digital Library

[15]

X. Chen, H. Fang, T. Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015).

[16]

T. B. Dinesh, S Uskudarli, S. Sastry, D. Aggarwal, and V. Choppella. 2012. Alipi: A framework for re-narrating web pages. In Proceedings of the International Cross-Disciplinary Conference on Web Accessibility.

Digital Library

[17]

J. Fraser and C. Gutwin. 2000. A framework of assistive pointers for low vision users. In ACM SIGACCESS conference on Computers and accessibility (ASSETS). 9--16.

Digital Library

[18]

Batya Friedman, Peter H Kahn, Alan Borning, and Alina Huldtgren. 2013. Value sensitive design and information systems. In Early engagement and new technologies: Opening up the laboratory. Springer, 55--95.

[19]

C. P. Gharpure and V. A. Kulyukin. 2008. Robot-assisted shopping for the blind: issues in spatial cognition and product selection. Intelligent Service Robotics 1, 3 (2008), 237--251.

[20]

D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3608--3617.

[21]

S. Harper and A. Q. Chen. 2012. Web accessibility guidelines. Web accessibility guidelines 15, 1 (2012), 61--88.

Digital Library

[22]

C. Harrison and A. K. Dey. 2008. Lean and zoom: proximity-aware user interface and content magnification. In ACM Conference on Human Factors in Computing Systems (CHI). 507--510.

Digital Library

[23]

T. Hironobu, S. Kawanaka, M. Kobayashi, T. Itoh, and C. Asakawa. 2008. Social Accessibility: Achieving Accessibility through Collaborative Metadata Authoring. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). 193--200.

Digital Library

[24]

M. Jas and D. Parikh. 2015. Image Specificity. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2727--2736.

[25]

H. Jethava, S. Zafar, and M. Saini. 2014. Electronic Shopping Cart Facility for Blind People Using USB Firmware. In International Journal of Emerging Technology and Advanced Engineering, Vol. 4. 647--651.

[26]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the 22Nd ACM International Conference on Multimedia (MM '14).

Digital Library

[27]

Y. Jing, D. Liu, D. Kislyuk, A. Zhai, J. Xu, J. Donahue, and S. Tavel. 2015. Visual Search at Pinterest. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1889--1898.

Digital Library

[28]

H. Kacorri, K. M. Kitani, J. P. Bigham, and C. Asakawa. 2017. People with Visual Impairment Training Personal Object Recognizers: Feasibility and Challenges. In ACM Conference on Human Factors in Computing Systems (CHI). ACM, 5839--5849.

Digital Library

[29]

S. Kesh, S. V. Venkateshwar, and A. U. 2017. Shopping By Blind People: Detection of Interactions in Ambient Assisted Living Environments using RFID. In International Journal of Wireless Communications and Networking Technologies, Vol. 6. 7--11.

[30]

D. Kislyuk, Y. Liu, D. Liu, E. Tzeng, and Y. Jing. 2015. Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest. arXiv preprint arXiv:1511.04003. (2015).

[31]

R.L. Kline and E.P. Glinert. 1995. Improving GUI accessibility for people with low vision. In ACM Conference on Human Factors in Computing Systems (CHI). 114--121.

Digital Library

[32]

A. Kovashka, D. Parikh, and K. Grauman. 2012. Whittlesearch: Image search with relative attribute feedback. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2973--2980.

Digital Library

[33]

W. S. Lasecki, R. Kushalnagar, and J. P. Bigham. 2014. Legion Scribe: Real-time Captioning by Non-experts. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). ACM, 303--304.

Digital Library

[34]

Xiaodan Liang, Chunyan Xu, Xiaohui Shen, Jianchao Yang, Si Liu, Jinhui Tang, Liang Lin, and Shuicheng Yan. 2015. Human Parsing With Contextualized Convolutional Neural Network. In The IEEE International Conference on Computer Vision (ICCV).

Digital Library

[35]

H. MacLeod, C. L. Bennett, M. R. Morris, and E. Cutrell. 2017. Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images. In ACM Conference on Human Factors in Computing Systems (CHI). ACM, 5988--5999.

Digital Library

[36]

N. Paisios, L. Subramanian, and A. Rubinsteyn. 2012. Choosing which clothes to wear confidently: A tool for pattern matching. In Workshop on Frontiers in Accessibility for Pervasive Computing. ACM.

[37]

H. Petrie and O. Kheir. 2007. The Relationship between Accessibility and Usability of Websites. In ACM Conference on Human Factors in Computing Systems (CHI). 397--406.

Digital Library

[38]

C. Power, A. P. Freire, H. Petrie, and D. Swallow. 2012. Guidelines are Only Half of the Story: Accessibility Problems Encountered by Blind Users on the Web. In ACM Conference on Human Factors in Computing Systems (CHI). 433--442.

Digital Library

[39]

G. VRJ Prasad, S. Chimalakonda, V. Choppella, and Y R. Reddy. 2017. An Aspect Oriented Approach for Renarrating Web Content. In ACM Innovations in Software Engineering Conference. 56--65.

Digital Library

[40]

E. Salisbury, E. Kamar, and M. Morris Ringel. 2017. Toward Scalable Social Alt Text: Conversational Crowdsourcing as a Tool for Refining Vision-to-Language Technology for the Blind. (2017).

[41]

M. D. Smucker, J. Allan, and B. Carterette. 2007. A Comparison of Statistical Significance Tests for Information Retrieval Evaluation. In ACM conference on Conference on Information and Knowledge Management. 623--632.

Digital Library

[42]

L. Stearns, V. DeSouza, J. Yin, L. Findlater, and J. E. Froehlich. 2017. Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS).

Digital Library

[43]

Anselm Strauss and Juliet M Corbin. 1990. Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc.

[44]

S. Szpiro, Y. Zhao, and S. Azenkot. 2016. Finding a store, searching for a product: a study of daily challenges of low vision people. In ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 61--72.

Digital Library

[45]

L. von Ahn, S. Ginosar, M. Kedia, R. Liu, and M. Blum. 2006. Improving Accessibility of the Web with a Computer Game. In SIGCHI Conference on Human Factors in computing systems (CHI). 79--82.

Digital Library

[46]

M. A Williams, A. Hurst, and S. K. Kane. 2013. Pray Before You Step Out: Describing Personal and Situational Blind Navigation Behaviors. In ACM SIGACCESS Conference on Computers and Accessibility (ASSETS).

Digital Library

[47]

T. Winlock, E. Christiansen, and S. Belongie. 2010. Toward real-time grocery detection for the visually impaired. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, 49--56.

[48]

S. Wu, J. Wieland, O. Farivar, and J. Schiller. 2017. Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service. In CSCW. 1180--1992.

Digital Library

[49]

Y. Yesilada. 2011. Web page segmentation: A review. Northern Cyprus Campus (2011).

[50]

S. Yuan, Y. Tian, and A. Arditi. 2011. Clothing matching for visually impaired persons. Technology and disability 23, 2 (2011), 75--85.

[51]

A. Zhai, D. Kislyuk, Y. Jing, M. Feng, E. Tzeng, J. Donahue, Y. L. Du, and T. Darrell. 2017. Visual Discovery at Pinterest. arXiv preprint arXiv:1702.04680. (2017).

[52]

Y. Zhao, S. Szpiro, J. Knighten, and S. Azenkot. 2016. CueSee: Exploring Visual Cues for People with Low Vision to Facilitate a Visual Search Task. In ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp). 73--84.

Digital Library

[53]

Y. Zhong, W. S. Lasecki, E. Brady, and J. P. Bigham. 2015. Regionspeak: Quick comprehensive spatial descriptions of complex images for blind users. In ACM Conference on Human Factors in Computing Systems (CHI). 2353--2362.

Digital Library

Cited By

Alsabhi R(2025)Tactile and sensory fashion garments for the visually impaired-challenges and opportunitiesInternational Journal of Fashion Design, Technology and Education10.1080/17543266.2024.2448243(1-9)Online publication date: 17-Jan-2025
https://doi.org/10.1080/17543266.2024.2448243
Benny Thadikaran GKumar Singh S(2024)Fostering Inclusion in Digital Marketplace: Vistas into the Online Shopping Experiences of Consumers with Visual Impairment in IndiaOrganizations and Markets in Emerging Economies10.15388/omee.2024.15.515:1(30)(90-108)Online publication date: 29-May-2024
https://doi.org/10.15388/omee.2024.15.5
Kamikubo RKacorri HAsakawa C(2024)"We are at the mercy of others' opinion": Supporting Blind People in Recreational Window Shopping with AI-infused TechnologyProceedings of the 21st International Web for All Conference10.1145/3677846.3677860(45-58)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677860
Show More Cited By

Recommendations

Misfitting With AI: How Blind People Verify and Contest AI Errors
ASSETS '24: Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility

Blind people use artificial intelligence-enabled visual assistance technologies (AI VAT) to gain visual access in their everyday lives, but these technologies are embedded with errors that may be difficult to verify non-visually. Previous studies have ...
Developing accessibility multimedia services: the case of EasyTV
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments

People with various degrees of disabilities (e.g., visually or hearing impaired) often find it difficult to access mainstream products and services and thus they are excluded from enjoying audio-visual services on an equal basis as people without ...
Menu controller: making existing software more accessible for people with motor impairments
PETRA '11: Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments

Menu Controller was developed to make existing software more accessible for people with severe motor impairments, especially individuals who use mouse-replacement input systems. Windows applications have menus that are difficult to access by users with ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

October 2018

508 pages

ISBN:9781450356503

DOI:10.1145/3234695

General Chair:
Faustina Hwang
University of Reading, UK
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
David Flatla
University of Guelph, Canada

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation
Amazon Research Award

Conference

ASSETS '18

Sponsor:

SIGACCESS

ASSETS '18: The 20th International ACM SIGACCESS Conference on Computers and Accessibility

October 22 - 24, 2018

Galway, Ireland

Acceptance Rates

ASSETS '18 Paper Acceptance Rate 28 of 108 submissions, 26%;

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,833
Total Downloads

Downloads (Last 12 months)464
Downloads (Last 6 weeks)42

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alsabhi R(2025)Tactile and sensory fashion garments for the visually impaired-challenges and opportunitiesInternational Journal of Fashion Design, Technology and Education10.1080/17543266.2024.2448243(1-9)Online publication date: 17-Jan-2025
https://doi.org/10.1080/17543266.2024.2448243
Benny Thadikaran GKumar Singh S(2024)Fostering Inclusion in Digital Marketplace: Vistas into the Online Shopping Experiences of Consumers with Visual Impairment in IndiaOrganizations and Markets in Emerging Economies10.15388/omee.2024.15.515:1(30)(90-108)Online publication date: 29-May-2024
https://doi.org/10.15388/omee.2024.15.5
Kamikubo RKacorri HAsakawa C(2024)"We are at the mercy of others' opinion": Supporting Blind People in Recreational Window Shopping with AI-infused TechnologyProceedings of the 21st International Web for All Conference10.1145/3677846.3677860(45-58)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3677846.3677860
Park SJung HRyskeldiev BLee JLee GJeong HChun MTeramoto K(2024)Toward Closed-Domain Conversational Item Listing Assistant for Improvement of Experiences of Older Adults in Customer-to-Customer (C2C) MarketplacesProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688522(1-5)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3688522
Gubbi Mohanbabu APavel A(2024)Context-Aware Image Descriptions for Web AccessibilityProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675658(1-17)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675658
Wang R(2024)Netizen A11y: Engaging Internet Users in Making Visual Media AccessibleCompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645247(159-162)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645247
Kim SLee SKim KOh U(2024)Utilizing a Dense Video Captioning Technique for Generating Image Descriptions of Comics for People with Visual ImpairmentsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645154(750-760)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645154
Van Daele TIyer AZhang YDerry JHuh MPavel A(2024)Making Short-Form Videos Accessible with Hierarchical Video SummariesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642839(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642839
Das MFiannaca AMorris MKane SBennett C(2024)From Provenance to Aberrations: Image Creator and Screen Reader User Perspectives on Alt Text for AI-Generated ImagesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642325(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642325
Reynolds JNagesh CGurari D(2024)Salient Object Detection for Images Taken by People With Vision Impairments2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00833(8507-8516)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00833
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten