research-article

Predictive web automation assistant for people with vision impairments

Authors:

Yevgen Borodin,

I.V. RamakrishnanAuthors Info & Claims

WWW '13: Proceedings of the 22nd international conference on World Wide Web

Pages 1031 - 1040

https://doi.org/10.1145/2488388.2488478

Published: 13 May 2013 Publication History

Abstract

The Web is far less usable and accessible for people with vision impairments than it is for sighted people. Web automation, a process of automating browsing actions on behalf of the user, has the potential to bridge the divide between the ways sighted and people with vision impairment access the Web; specifically, it can enable the latter to breeze through web browsing tasks that beforehand were slow, hard, or even impossible to accomplish. Typical web automation requires that the user record a macro, a sequence of browsing steps, so that these steps can be automated in the future by replaying the macro. However, for people with vision impairment, automation with macros is not usable.

In this paper, we propose a novel model-based approach that facilitates web automation without having to either record or replay macros. Using the past browsing history and the current web page as the browsing context, the proposed model can predict the most probable browsing actions that the user can do. The model construction is "unsupervised". More importantly, the model is continuously and incrementally updated as history evolves, thereby, ensuring the predictions are not "outdated".

We also describe a novel interface that lets the user focus on the objects associated with the most probable predicted browsing steps (e.g., clicking links and filling out forms), and facilitates automatic execution of the selected steps. A study with 19 blind participants showed that the proposed approach dramatically reduced the interaction time needed to accomplish typical browsing tasks, and the user interface was perceived to be much more usable than the standard screen-reading interfaces.

References

[1]

Freedom Scientific: JAWS screen-reader. http://www.freedomscientific.com/.

[2]

GW Micro: Window-Eyes screen-reader. http://www.gwmicro.com/.

[3]

NonVisual Desktop Access: NVDA screen-reader. http://www.nvda-project.org/.

[4]

World Wide Web consortium (W3C): XML path language (XPath), 1999. http://www.w3.org/TR/xpath.

[5]

IVONA software: IVONA multi-lingual speech synthesis system, 2005. http://www.ivona.com/.

[6]

Bangor, A., Kortum, P. T., and Miller, J. T. An empirical evaluation of the System Usability Scale. Int. J. Hum. Comput. Interaction (2008), 574--594.

[7]

Bian, J., Dong, A., He, X., Reddy, S., and Chang, Y. User action interpretation for online content optimization. Knowledge and Data Engineering, IEEE Transactions on PP, 99 (2012), 1.

[8]

Bigham, J. P., Lau, T., and Nichols, J. Trailblazer: enabling blind users to blaze trails through the web. In Proceedings of the 14th international conference on Intelligent user interfaces, IUI '09, ACM (New York, NY, USA, 2009), 177--186.

Digital Library

[9]

Bolin, M., Webber, M., Rha, P., Wilson, T., and Miller, R. C. Automation and customization of rendered web pages. In Proceedings of the 18th annual ACM symposium on User interface software and technology, UIST '05, ACM (New York, NY, USA, 2005), 163--172.

Digital Library

[10]

Bose, A., Beemanapalli, K., Srivastava, J., and Sahar, S. Incorporating concept hierarchies into usage mining based recommendations. In Proceedings of the 8th Knowledge discovery on the web international conference on Advances in web mining and web usage analysis, WebKDD'06, Springer-Verlag (Berlin, Heidelberg, 2007), 110--126.

Digital Library

[11]

Brooke, J. SUS: A quick and dirty usability scale. In Usability evaluation in industry, P. W. Jordan, A. Thomas, B. Weerdmeester, and I. McClelland, Eds., Taylor and Francis (1996).

[12]

Cypher, A., Halbert, D. C., Kurlander, D., Lieberman, H., Maulsby, D., Myers, B. A., and Turransky, A., Eds. Watch what I do: programming by demonstration. MIT Press, Cambridge, MA, USA, 1993.

Digital Library

[13]

Domènech, J., de la Ossa, B., Sahuquillo, J., Gil, J.-A., and Pont, A. A taxonomy of web prediction algorithms. Expert Syst. Appl. 39, 9 (2012), 8496--8502.

Digital Library

[14]

Findlater, L., and Gajos, K. Z. Design Space and Evaluation Challenges of Adaptive Graphical User Interfaces. AI Magazine 30, 4 (2009), 68--73.

[15]

Huang, Z., Eliens, A., van Ballegooij, A., and de Bra, P. A taxonomy of web agents. In Database and Expert Systems Applications, 2000. Proceedings. 11th International Workshop on (2000), 765 --769.

Digital Library

[16]

Hupp, D., and Miller, R. C. Smart bookmarks: automatic retroactive macro recording on the web. In Proceedings of the 20th annual ACM symposium on User interface software and technology, UIST '07, ACM (New York, NY, USA, 2007), 81--90.

Digital Library

[17]

Leshed, G., Haber, E. M., Matthews, T., and Lau, T. Coscripter: automating & sharing how-to knowledge in the enterprise. In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI '08, ACM (New York, NY, USA, 2008), 1719--1728.

Digital Library

[18]

Lieberman, H. Autonomous interface agents. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '97, ACM (New York, NY, USA, 1997), 67--74.

Digital Library

[19]

Mahmud, J., Borodin, Y., Ramakrishnan, I. V., and Ramakrishnan, C. R. Automated construction of web accessibility models from transaction click-streams. In Proceedings of the 18th international conference on World Wide Web, WWW '09, ACM (New York, NY, USA, 2009), 871--880.

Digital Library

[20]

Mahmud, J., Sun, Z., Mukherjee, S., and Ramakrishnan, I. Abstract web transactions on handhelds with less tears. In Proceedings of the Workshop MobEA IV - Empowering the Mobile Web (2006).

[21]

Montoto, P., Pan, A., Raposo, J., Bellas, F., and López, J. Automating navigation sequences in ajax websites. In Proceedings of the 9th International Conference on Web Engineering, ICWE '09, Springer-Verlag (Berlin, Heidelberg, 2009), 166--180.

Digital Library

[22]

Myers, E. W., and Miller, W. Optimal alignments in linear space. CABIOS 4 (1988), 11--17.

[23]

Padmanabhan, V. N., and Mogul, J. C. Using predictive prefetching to improve World Wide Web latency. SIGCOMM Comput. Commun. Rev. 26, 3 (July 1996), 22--36.

Digital Library

[24]

Puzis, Y. Accessible web automation interface: a user study. In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility, ASSETS '12, ACM (New York, NY, USA, 2012), 291--292.

Digital Library

[25]

Puzis, Y. An interface agent for non-visual, accessible web automation. In Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology, UIST Adjunct Proceedings '12, ACM (New York, NY, USA, 2012), 55--58.

Digital Library

[26]

Puzis, Y., Borodin, E., Ahmed, F., Melnyk, V., and Ramakrishnan, I. V. Guidelines for an accessible web automation interface. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility, ASSETS '11, ACM (New York, NY, USA, 2011), 249--250.

Digital Library

[27]

Puzis, Y., Borodin, Y., Ahmed, F., and Ramakrishnan, I. V. An intuitive accessible web automation user interface. In Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A '12, ACM (New York, NY, USA, 2012), 41:1--41:4.

Digital Library

[28]

Sun, Z., Mahmud, J., Ramakrishnan, I. V., and Mukherjee, S. Model-directed web transactions under constrained modalities. ACM Trans. Web 1, 3 (Sept. 2007).

Digital Library

[29]

Vanesa Aciar, S., Serarols-Tarres, C., Royo-Vela, M., and De la Rosa i Esteva, J. L. Increasing effectiveness in e-commerce: recommendations applying intelligent agents. International Journal of Business and Systems Research 1, 1 (01 2007), 81--97.

[30]

Waterman, M. S., and Smith, T. F. Identification of common molecular subsequences. J. Mol. Biol. 147 (1981), 195--197.

Cited By

Liu JWang ZAyyubi HYou HThomas CSun RChang SChang KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless PredictionsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681622(8402-8411)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681622
Islam MSojib NKabir IAmit AAmin MBillah S(2024)Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual InteractionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676396(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676396
Ferdous JLee HJayarathna SAshok V(2023)Enabling Efficient Web Data-Record Interaction for People with Visual Impairments via Proxy InterfacesACM Transactions on Interactive Intelligent Systems10.1145/357936413:3(1-27)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3579364
Show More Cited By

Index Terms

Predictive web automation assistant for people with vision impairments
1. Applied computing
  1. Document management and text processing
    1. Document preparation
      1. Multi / mixed media creation
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Hypertext / hypermedia

Recommendations

Web Screen Reading Automation Assistance Using Semantic Abstraction
IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces

A screen reader's sequential press-and-listen interface makes for an unsatisfactory and often times painful web-browsing experience for blind people. To help alleviate this situation, we introduce Web Screen Reading Automation Assistant (SRAA) for ...
Predictive, accessible web automation: a longitudinal study
W4A '14: Proceedings of the 11th Web for All Conference

The problem of efficient, usable non-visual web access remains inadequately solved, despite its critical importance to a significant number of visually-impaired existing and potential Web users. Web automation, a process of automating browsing actions ...
Complexities of practical web automation
W4A '15: Proceedings of the 12th International Web for All Conference

Web automation, a process of automating browsing actions on behalf of the user, has the potential to bridge the divide between the ways visually-impaired and sighted people access the Web. The proliferation of portable small-screen devices (smartphones, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

WWW '13: Proceedings of the 22nd international conference on World Wide Web

May 2013

1628 pages

ISBN:9781450320351

DOI:10.1145/2488388

General Chairs:
Daniel Schwabe
PUC-Rio - Brazil
,
Virgílio Almeida
UFMG - Brazil
,
Hartmut Glaser
CGI.br - Brazil
,
Program Chairs:
Ricardo Baeza-Yates
Yahoo! Labs - Spain & Chile
,
Sue Moon
KAIST - South Korea

Copyright © 2013 Copyright is held by the International World Wide Web Conference Committee (IW3C2).

Sponsors

NICBR: Nucleo de Informatcao e Coordenacao do Ponto BR
CGIBR: Comite Gestor da Internet no Brazil

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WWW '13

Sponsor:

NICBR
CGIBR

WWW '13: 22nd International World Wide Web Conference

May 13 - 17, 2013

Rio de Janeiro, Brazil

Acceptance Rates

WWW '13 Paper Acceptance Rate 125 of 831 submissions, 15%;

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
356
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu JWang ZAyyubi HYou HThomas CSun RChang SChang KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless PredictionsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681622(8402-8411)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681622
Islam MSojib NKabir IAmit AAmin MBillah S(2024)Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual InteractionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676396(1-20)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676396
Ferdous JLee HJayarathna SAshok V(2023)Enabling Efficient Web Data-Record Interaction for People with Visual Impairments via Proxy InterfacesACM Transactions on Interactive Intelligent Systems10.1145/357936413:3(1-27)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3579364
Leschke NKirsten FPallas FGrünewald E(2023)Streamlining Personal Data Access Requests: From Obstructive Procedures to Automated Web WorkflowsWeb Engineering10.1007/978-3-031-34444-2_9(111-125)Online publication date: 16-Jun-2023
https://doi.org/10.1007/978-3-031-34444-2_9
Barbosa NHayes JKaushik SWang Y(2022)“Every Website Is a Puzzle!”: Facilitating Access to Common Website Features for People with Visual ImpairmentsACM Transactions on Accessible Computing10.1145/351903215:3(1-35)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.1145/3519032
Ferdous JLee HJayarathna SAshok V(2022)InSupport: Proxy Interface for Enabling Efficient Non-Visual Interaction with Web Data RecordsProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511126(49-62)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511126
Ferdous JUddin SAshok VHung CHong JBechini ASong E(2021)Semantic table-of-contents for efficient web screen readingProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3442066(1941-1949)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3442066
Pina ABaez MDaniel F(2021)Bringing Cognitive Augmentation to Web Browsing AccessibilityService-Oriented Computing – ICSOC 2020 Workshops10.1007/978-3-030-76352-7_37(395-407)Online publication date: 30-May-2021
https://doi.org/10.1007/978-3-030-76352-7_37
Sojib NBillah SAmin MIqbal SMacLean KChevalier FMueller S(2020)Towards Multi-Wheel Input Device for Non-Visual InteractionAdjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379350.3416168(7-9)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379350.3416168
Lee HUddin SAshok V(2020)iTOC: Enabling Efficient Non-Visual Interaction with Long Web Documents2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC42975.2020.9282972(3799-3806)Online publication date: 11-Oct-2020
https://doi.org/10.1109/SMC42975.2020.9282972
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents