research-article

Open access

Predicting Human Performance in Vertical Menu Selection Using Deep Learning

Authors:

Gilles BaillyAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 29, Pages 1 - 7

https://doi.org/10.1145/3173574.3173603

Published: 19 April 2018 Publication History

Abstract

Predicting human performance in interaction tasks allows designers or developers to understand the expected performance of a target interface without actually testing it with real users. In this work, we present a deep neural net to model and predict human performance in performing a sequence of UI tasks. In particular, we focus on a dominant class of tasks, i.e., target selection from a vertical list or menu. We experimented with our deep neural net using a public dataset collected from a desktop laboratory environment and a dataset collected from hundreds of touchscreen smartphone users via crowdsourcing. Our model significantly outperformed previous methods on these datasets. Importantly, our method, as a deep model, can easily incorporate additional UI attributes such as visual appearance and content semantics without changing model architectures. By understanding about how a deep learning model learns from human behaviors, our approach can be seen as a vehicle to discover new patterns about human behaviors to advance analytical modeling.

References

[1]

Bahdanau, D., Cho, K., and Bengio, Y., Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 15: International Conference on Learning Representations, 2015.

[2]

Bailly, G., Oulasvirta, A., Brumby, D.P., and Howes, A. Model of visual search and selection time in linear menus. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2014. 3865--3874.

Digital Library

[3]

Bi, X., Li, Y., and Zhai, S. FFitts Law: Modeling Finger Touch With Fitts' Law. CHI 2013: ACM Conference on Human Factors in Computing Systems. 2013.

Digital Library

[4]

Byrne, M.D., ACT-R/PM and menu selection: Applying a cognitive architecture to HCI. International Journal of Human-Computer Studies, 2001. 55(1): p. 41--84.

Digital Library

[5]

Card, S.K., Moran, T.P., and Newell, A., The keystroke-level model for user performance time with interactive systems. Communications of the ACM, 1980. 23(7): p. 396--410.

Digital Library

[6]

Chen, X., Bailly, G., Brumby, D.P., Oulasvirta, A., and Howes, A. The Emergence of Interactive Behavior: A Model of Rational Menu Search. CHI'15: ACM Conference of Human Factors in Computing Systems. 2015. 4217--4226.

Digital Library

[7]

Cockburn, A., Gutwin, C., and Greenberg, S. A predictive model of menu performance. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2007. 627--636

Digital Library

[8]

Fitts, P.M., The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement. Journal of Experimental Psychology, 1954. 121(3): p. 381--391.

[9]

Graves, A., Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence. 2012: Springer.

[10]

Halverson, T. and Hornof, A.J., A Computational Model of "Active Vision" for Visual Search in Human--Computer Interaction. Human-Computer Interaction, 2011. 26(4): p. 285--314.

Digital Library

[11]

Hick, W.E., On the rate of gain of information. Journal of Experimental Psychology, 1952. 4(1): p. 11--26.

[12]

Hochreiter, S. and Schmidhuber, J.u., Long Short-Term Memory. Neural Computation, 1997. 9(8): p. 1735--1780.

Digital Library

[13]

Kaptelinin, V. Item recognition in menu selection: the effect of practice. CHI '93 INTERACT '93 and CHI '93 Conference Companion on Human Factors in Computing Systems. 1993. 183--184.

Digital Library

[14]

Kieras, D.E. and Meyer, D.E., An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Hum.-Comput. Interact., 1997. 12(4): p. 391--438.

Digital Library

[15]

LeCun, Y., Bengio, Y., and Hinton, G., Deep learning. Nature, 2015(521): p. 436--444.

[16]

MacKenzie, I.S. and Buxton, W. Extending Fitts' law to two-dimensional tasks. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1992. 219--226.

Digital Library

[17]

Martín Abadi, P.B., Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng. TensorFlow: A system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 2016. 265--283.

Digital Library

[18]

Nair, V. and Hinton, G.E. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML: Proceedings of the 27 th International Conference on Machine Learning. 2010.

Digital Library

[19]

Näsänena, R., Ojanpääa, H., and Kojob, I., Effect of stimulus contrast on performance and eye movements in visual search. Vision Research, 2001. 41(14): p. 1817--1824.

[20]

Pennington, J., Socher, R., and Manning, C.D. GloVe: Global Vectors for Word Representation. 2014.

[21]

Sears, A. and Shneiderman, B., Split menus: effectively using selection frequency to organize menus. ACM Trans. Comput.-Hum. Interact., 1994. 1(1): p. 27--51.

Digital Library

Cited By

Yousefi ZVuong TAlGhossein MRuotsalo TJaccuci GKaski S(2024)Entity Footprinting: Modeling Contextual User States via Digital Activity MonitoringACM Transactions on Interactive Intelligent Systems10.1145/3643893Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3643893
Lorenz MAmorim TDey DSadeghi MEbel P(2024)Computational Models for In-Vehicle User Interface Design: A Systematic Literature ReviewProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675735(204-215)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675735
Gardey JGrigera JRodriguez AGarrido AHong JPark J(2024)UX-Analyzer: Visualizing the interaction effort for web analyticsProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3636013(1774-1780)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3636013
Show More Cited By

Index Terms

Predicting Human Performance in Vertical Menu Selection Using Deep Learning
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
2. Human-centered computing

Recommendations

New Results for Prediction of Chaotic Systems Using Deep Recurrent Neural Networks
Abstract
Prediction of nonlinear and dynamic systems is a challenging task, however with the aid of machine learning techniques, particularly neural networks, is now possible to accomplish this objective. Most common neural networks used are the multilayer ...
Document Modeling with Hierarchical Deep Learning Approach for Sentiment Classification
ICDSP '18: Proceedings of the 2nd International Conference on Digital Signal Processing

Sentiment analysis has recently been considered as most active research field in NLP domain. Deep learning is a growing trend of machine learning due to its automatic learning capability with impressive results across different NLP task. In this paper a ...
Benchmarking deep learning techniques for face recognition
Highlights
- Training networks for face recognition is very complex and time-consuming.
- ...
Abstract
Recent progresses in Convolutional Neural Networks (CNNs) and GPUs have greatly advanced the state-of-the-art performance for face recognition. However, training CNNs for face recognition is complex and time-consuming. Multiple factors ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2018

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
2,191
Total Downloads

Downloads (Last 12 months)211
Downloads (Last 6 weeks)17

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yousefi ZVuong TAlGhossein MRuotsalo TJaccuci GKaski S(2024)Entity Footprinting: Modeling Contextual User States via Digital Activity MonitoringACM Transactions on Interactive Intelligent Systems10.1145/3643893Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3643893
Lorenz MAmorim TDey DSadeghi MEbel P(2024)Computational Models for In-Vehicle User Interface Design: A Systematic Literature ReviewProceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3640792.3675735(204-215)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3640792.3675735
Gardey JGrigera JRodriguez AGarrido AHong JPark J(2024)UX-Analyzer: Visualizing the interaction effort for web analyticsProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing10.1145/3605098.3636013(1774-1780)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1145/3605098.3636013
Lee LYau YHui P(2024)Perceived User Reachability in Mobile UIs Using Data Analytics and Machine LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2327199(1-24)Online publication date: 25-Mar-2024
https://doi.org/10.1080/10447318.2024.2327199
Yamanaka SUsuba H(2023)Tuning Endpoint-variability Parameters by Observed Error Rates to Obtain Better Prediction Accuracy of Pointing MissesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580746(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580746
Bailly GKhamassi MGirard B(2023)Computational Model of the Transition from Novice to Expert Interaction TechniquesACM Transactions on Computer-Human Interaction10.1145/350555730:5(1-33)Online publication date: 23-Sep-2023
https://dl.acm.org/doi/10.1145/3505557
Wang WGai WZang CBao XYang C(2023)Personalized Recommendation of User Interfaces Based on Foot Interaction2023 IEEE Smart World Congress (SWC)10.1109/SWC57546.2023.10448805(1-8)Online publication date: 28-Aug-2023
https://doi.org/10.1109/SWC57546.2023.10448805
Pourmemar MPoullis C(2023)Analysis of Hand Movement and Head Orientation in Hierarchical Menu Selection in Immersive AR2023 IEEE International Symposium on Multimedia (ISM)10.1109/ISM59092.2023.00052(270-275)Online publication date: 11-Dec-2023
https://doi.org/10.1109/ISM59092.2023.00052
Somaskantharajan SDegbelo AKrukar J(2023)An Exploratory Study of Models of Mobile Map User ExperienceEine explorative Studie über Modelle der Nutzererfahrung bei mobilen KartenKN - Journal of Cartography and Geographic Information10.1007/s42489-023-00136-873:2(127-146)Online publication date: 29-Apr-2023
https://doi.org/10.1007/s42489-023-00136-8
Ahmed ASalah AKhoriba GArafa T(2023)Localizing Non-functional Code Bugs in User Interfaces Using Deep Learning TechniquesModel and Data Engineering10.1007/978-3-031-49333-1_27(381-394)Online publication date: 22-Dec-2023
https://doi.org/10.1007/978-3-031-49333-1_27
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents