research-article

Spotting Visual Keywords from Temporal Sliding Windows

Authors:

Yue Yao,

Tianyu Wang,

Heming Du,

Liang Zheng,

Tom GedeonAuthors Info & Claims

ICMI '19: 2019 International Conference on Multimodal Interaction

Pages 536 - 539

https://doi.org/10.1145/3340555.3356101

Published: 14 October 2019 Publication History

Get Access

Abstract

Visual Keyword Spotting (KWS), as a newly proposed task deriving from visual speech recognition, has plenty of room for improvements. This paper details our Visual Keyword Spotting system used in the first Mandarin Audio-Visual Speech Recognition Challenge (MAVSR 2019). With the assumption that the vocabularies of target dataset are a subset of the vocabulary of the training set, we proposed a simple and scalable classification based strategy that achieves 19.0% mean average precision (mAP) on this challenge. Our method is based on the idea of using sliding windows to bridge between the word-level dataset and the sentence-level dataset, showing that a strong word level classifier can be directly used in building sentence embedding, thereby making it possible to build a KWS system.

References

[1]

Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2018. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence (2018).

Google Scholar

[2]

Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308.

Crossref

Google Scholar

[3]

Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar. 2018. Rethinking the faster r-cnn architecture for temporal action localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1130–1139.

Crossref

Google Scholar

[4]

Joon Son Chung and Andrew Zisserman. 2016. Lip reading in the wild. In Asian Conference on Computer Vision. Springer, 87–103.

Google Scholar

[5]

Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America 120, 5 (2006), 2421–2424.

Crossref

Google Scholar

[6]

Andrzej Czyzewski, Bozena Kostek, Piotr Bratoszewski, Jozef Kotus, and Marcin Szykulski. 2017. An audio-visual corpus for multimodal automatic speech recognition. Journal of Intelligent Information Systems 49, 2 (2017), 167–192.

Digital Library

Google Scholar

[7]

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2018. SlowFast Networks for Video Recognition. arXiv preprint arXiv:1812.03982(2018).

Google Scholar

[8]

Themos Stafylakis and Georgios Tzimiropoulos. 2017. Combining Residual Networks with LSTMs for Lipreading. Proc. Interspeech 2017(2017), 3652–3656.

Crossref

Google Scholar

[9]

Themos Stafylakis and Georgios Tzimiropoulos. 2018. Zero-Shot Keyword Spotting for Visual Speech Recognition In-the-wild. In European Conference on Computer Vision. Springer, 536–552.

Google Scholar

[10]

Huijuan Xu, Abir Das, and Kate Saenko. 2017. R-C3D: Region Convolutional 3D Network for Temporal Activity Detection. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 5794–5803.

Crossref

Google Scholar

[11]

Shuang Yang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang, Jingyun Xiao, Keyu Long, Shiguang Shan, and Xilin Chen. 2019. LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 1–8.

Digital Library

Google Scholar

Cited By

View all

Xiong SDai L(2024)Exploring Semi-Supervised, Subcategory Classification and Subwords Alignment for Visual Wake Word Spotting2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)10.1109/ICMEW63481.2024.10645411(1-6)Online publication date: 15-Jul-2024
https://doi.org/10.1109/ICMEW63481.2024.10645411
Rathipriya NMaheswari N(2024)A Comprehensive Review of Recent Advances in Deep Neural Networks for Lipreading With Sign Language RecognitionIEEE Access10.1109/ACCESS.2024.346396912(136846-136879)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3463969
S.J. PB. N(2023)Analyzing lower half facial gestures for lip reading applicationsComputer Vision and Image Understanding10.1016/j.cviu.2023.103738233:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.cviu.2023.103738
Show More Cited By

Recommendations

Zero-Shot Keyword Spotting for Visual Speech Recognition In-the-wild
Computer Vision – ECCV 2018
Abstract
Visual keyword spotting (KWS) is the problem of estimating whether a text query occurs in a given recording using only video information. This paper focuses on visual KWS for words unseen during training, a real-world, practical setting which so ...
A multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset
Abstract
Automatic lip reading has advanced significantly in recent years. However, these methods need large-scale datasets that are scarce for many low-resource languages. In this paper, we introduce a new multipurpose audio-visual dataset for Persian. ...
Highlights
- New multipurpose audiovisual dataset for Persian language.
- First large-scale lip-reading dataset in Persian.
- Dataset suitable for lip-reading, ASR and audio-visual speech recognition.
- A new technique for detecting visemes in ...
Lip tracking method for the system of audio-visual polish speech recognition
ICAISC'12: Proceedings of the 11th international conference on Artificial Intelligence and Soft Computing - Volume Part I

This paper proposes a method of tracking the lips in the system of audio-visual speech recognition. Presented methods consists of a face detector, face tracker, lip detector, lip tracker, and word classifier. In speech recognition systems, the audio ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICMI '19: 2019 International Conference on Multimodal Interaction

October 2019

601 pages

ISBN:9781450368605

DOI:10.1145/3340555

Editors:
Wen Gao
Peking University, China
,
Helen Mei Ling Meng
Chinese University of Hong Kong, China
,
Matthew Turk
Toyota Technological Institute at Chicago, USA
,
Susan R. Fussell
Cornell University, USA
,
Björn Schuller
Imperial College London / University of Augsburg, UK
,
Yale Song
Microsoft Research, USA
,
Kai Yu
Shanghai Jiao Tong University, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
165
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Xiong SDai L(2024)Exploring Semi-Supervised, Subcategory Classification and Subwords Alignment for Visual Wake Word Spotting2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)10.1109/ICMEW63481.2024.10645411(1-6)Online publication date: 15-Jul-2024
https://doi.org/10.1109/ICMEW63481.2024.10645411
Rathipriya NMaheswari N(2024)A Comprehensive Review of Recent Advances in Deep Neural Networks for Lipreading With Sign Language RecognitionIEEE Access10.1109/ACCESS.2024.346396912(136846-136879)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3463969
S.J. PB. N(2023)Analyzing lower half facial gestures for lip reading applicationsComputer Vision and Image Understanding10.1016/j.cviu.2023.103738233:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.cviu.2023.103738
Sheng HYu XLi XGolzan M(2023)Context-Based Masking for Spontaneous Venous Pulsations DetectionAI 2023: Advances in Artificial Intelligence10.1007/978-981-99-8388-9_42(520-532)Online publication date: 27-Nov-2023
https://doi.org/10.1007/978-981-99-8388-9_42
Pu GWang H(2022)Review on research progress of machine lip readingThe Visual Computer10.1007/s00371-022-02511-439:7(3041-3057)Online publication date: 15-Jun-2022
https://doi.org/10.1007/s00371-022-02511-4
Albanie SVarol GMomeni LAfouras TChung JFox NZisserman A(2020)BSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing CuesComputer Vision – ECCV 202010.1007/978-3-030-58621-8_3(35-53)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58621-8_3

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

Zero-Shot Keyword Spotting for Visual Speech Recognition In-the-wild

A multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset

Lip tracking method for the system of audio-visual polish speech recognition

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations