poster

The Relation of Eye Gaze and Face Pose: Potential Impact on Speech Recognition

Authors:

Malcolm Slaney,

Andreas Stolcke,

Dilek Hakkani-TürAuthors Info & Claims

ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Pages 144 - 147

https://doi.org/10.1145/2663204.2663251

Published: 12 November 2014 Publication History

Get Access

Abstract

We are interested in using context to improve speech recognition and speech understanding. Knowing what the user is attending to visually helps us predict their utterances and thus makes speech recognition easier. Eye gaze is one way to access this signal, but is often unavailable (or expensive to gather) at longer distances. In this paper we look at joint eye-gaze and facial-pose information while users perform a speech reading task. We hypothesize, and verify experimentally, that the eyes lead, and then the face follows. Face pose might not be as fast, or as accurate a signal of visual attention as eye gaze, but based on experiments correlating eye gaze with speech recognition, we conclude that face pose provides useful information to bias a recognizer toward higher accuracy.

References

[1]

1. N. J. Cooke and M. Russell. Gaze-contingent automatic speech recognition. In Signal Processing, pages 369--380, 2008.

Google Scholar

[2]

2. L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, and J. Williams. Recent advances in deep learning for speech research at Microsoft. In Proceedings of the 2013 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2013.

Crossref

Google Scholar

[3]

3. J.-B. Huang, Q. Cai, Z. Liu, N. Ahuja, and Z. Zhang. Towards accurate and robust cross-ratio based gaze trackers through learning from simulation. In Proceedings of the Symposium on Eye Tracking Research and Applications, ETRA '14, pages 75--82, New York, NY, USA, 2014. ACM.

Digital Library

Google Scholar

[4]

4. D. Klakow. Log-linear interpolation of language models. In Proceedings of the International Conference on Spoken-Language Processing (ICSLP), page 1695, 1998.

Google Scholar

[5]

5. M. Ostendorf, A. Kannan, S. Austin, O. Kimball, R. Schwartz, and J. R. Rohlicek. Integration of diverse recognition methodologies through reevaluation of n-best sentence hypotheses. In Proceedings of the workshop on Speech and Natural Language (HLT '91), pages 83--87. Association for Computational Linguistics, 1991.

Digital Library

Google Scholar

[6]

6. M. Slaney, R. Rajen, A. Stolcke, and P. Parthasarathy. Gaze enhanced speech recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.

Crossref

Google Scholar

Cited By

View all

Khan ANewn JBailey JVelloso E(2022)Integrating Gaze and Speech for Enabling Implicit InteractionsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502134(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502134
Holman BAnwar ASingh ATec MHart JStone P(2021)Watch Where You’re Going! Gaze and Head Orientation as Predictors for Social Robot Navigation2021 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48506.2021.9561286(3553-3559)Online publication date: 30-May-2021
https://doi.org/10.1109/ICRA48506.2021.9561286
Rothe SPothmann PDrewe HHussmann H(2019)Interaction Techniques for Cinematic Virtual Reality2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR.2019.8798189(1733-1737)Online publication date: Mar-2019
https://doi.org/10.1109/VR.2019.8798189
Show More Cited By

Index Terms

The Relation of Eye Gaze and Face Pose: Potential Impact on Speech Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing

Recommendations

Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze
ICMI '19: 2019 International Conference on Multimodal Interaction

Continuous emotion recognition is of great significance in affective computing and human-computer interaction. Most of existing methods for video based continuous emotion recognition utilize facial expression. However, besides facial expression, other ...
EFG-Net: A Unified Framework for Estimating Eye Gaze and Face Gaze Simultaneously
Pattern Recognition and Computer Vision
Abstract
Gaze is of vital importance for understanding human purpose and intention. Recent works have gained tremendous progress in appearance-based gaze estimation. However, all these works deal with eye gaze estimation or face gaze estimation separately, ...
Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality

Humans perform gaze shifts naturally through a combination of eye, head and body movements. Although gaze has been long studied as input modality for interaction, this has previously ignored the coordination of the eyes, head and body. This article ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

November 2014

558 pages

ISBN:9781450328852

DOI:10.1145/2663204

General Chairs:
Albert Ali Salah
Boğaziçi University, Turkey
,
Jeffrey Cohn
University of Pittsburgh, USA
,
Björn Schuller
University of Passau, Germany and Imperial College London, UK
,
Program Chairs:
Oya Aran
Idiap Research Institute, Switzerland
,
Louis-Philippe Morency
University of Southern California, USA
,
Philip R. Cohen
Adapx, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

ICMI '14

Sponsor:

SIGCHI

ICMI '14: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2014

Istanbul, Turkey

Acceptance Rates

ICMI '14 Paper Acceptance Rate 51 of 127 submissions, 40%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
176
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Khan ANewn JBailey JVelloso E(2022)Integrating Gaze and Speech for Enabling Implicit InteractionsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502134(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502134
Holman BAnwar ASingh ATec MHart JStone P(2021)Watch Where You’re Going! Gaze and Head Orientation as Predictors for Social Robot Navigation2021 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48506.2021.9561286(3553-3559)Online publication date: 30-May-2021
https://doi.org/10.1109/ICRA48506.2021.9561286
Rothe SPothmann PDrewe HHussmann H(2019)Interaction Techniques for Cinematic Virtual Reality2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR.2019.8798189(1733-1737)Online publication date: Mar-2019
https://doi.org/10.1109/VR.2019.8798189
Rothe SAlthammer FKhamis M(2018)GazeRecallProceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia10.1145/3282894.3282903(115-119)Online publication date: 25-Nov-2018
https://dl.acm.org/doi/10.1145/3282894.3282903
Gödde MGabler FSiegmund DBraun A(2018)Cinematic Narration in VR – Rethinking Film Conventions for 360 DegreesVirtual, Augmented and Mixed Reality: Applications in Health, Cultural Heritage, and Industry10.1007/978-3-319-91584-5_15(184-201)Online publication date: 2-Jun-2018
https://doi.org/10.1007/978-3-319-91584-5_15
Szirtes GOrozco JPetrs ISzolgay DUtasi kCohn J(2017)Behavioral cues help predict impact of advertising on future salesImage and Vision Computing10.1016/j.imavis.2017.03.00265:C(49-57)Online publication date: 1-Sep-2017
https://dl.acm.org/doi/10.1016/j.imavis.2017.03.002
Haider FLuz SCampbell N(2016)Data Collection and Synchronisation: Towards a Multiperspective Multimodal Dialogue System with Metacognitive AbilitiesDialogues with Social Robots10.1007/978-981-10-2585-3_19(245-256)Online publication date: 25-Dec-2016
https://doi.org/10.1007/978-981-10-2585-3_19
Wagner AGray J(2015)An Empirical Evaluation of a Vocal User Interface for Programming by VoiceInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.20150701048:2(47-63)Online publication date: 1-Jul-2015
https://doi.org/10.4018/IJITSA.2015070104
Prokofieva ASlaney MHakkani-Tur D(2015)Probabilistic features for connecting eye gaze to spoken language understanding2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2015.7178985(5311-5315)Online publication date: Apr-2015
https://doi.org/10.1109/ICASSP.2015.7178985
Acartürk CFreitas JFal MDias M(2015)Elderly Speech-Gaze InteractionUniversal Access in Human-Computer Interaction. Access to Today's Technologies10.1007/978-3-319-20678-3_1(3-12)Online publication date: 18-Jul-2015
https://doi.org/10.1007/978-3-319-20678-3_1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze

EFG-Net: A Unified Framework for Estimating Eye Gaze and Face Gaze Simultaneously

Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality