Abstract
Visual images synchronized with audio signals can provide user-friendly interface for man machine interactions. The visual speech can be represented as a sequence of visemes, which are the generic face images corresponding to particular sounds. We use HMMs (hidden Markov models) to convert audio signals to a sequence of visemes. In this paper, we compare two approaches in using HMMs. In the first approach, an HMM is trained for each triviseme which is a viseme with its left and right context, and the audio signals are directly recognized as a sequence of trivisemes. In the second approach, each triphone is modeled with an HMM, and a general triphone recognizer is used to produce a triphone sequence from the audio signals. The triviseme or triphone sequence is then converted to a viseme sequence. The performances of the two viseme recognition systems are evaluated on the TIMIT speech corpus.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Choi, K., Hwang, J.: Baum-Welch HMM inversion for audio-to-visual conversion, IEEE International Workshop on Multimedia Signal Processing, pp. 175–180, 1999.
Fisher, C.: Confusions among visually perceived consonants, Journal on Speech and Hearing Research, vol. 11, pp. 796–804, 1968.
Grant, K., Walden, B., Seitz, P.: Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration, Journal of Acoustic Society of America, vol. 103, pp. 2677–2690, 1998.
Morishima, S., Harashima, H.: A media conversion from speech to facial image for intelligent man-machine interface, IEEE Journal on selected areas in communications, vol. 9, no. 4, pp. 594–600, 1991.
Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition, Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989.
Rao, R., Chen, T.: Mersereau, R., Audio-to-visual conversion for multimedia communication, IEEE Transaction on Industrial Electronics, vol. 45, no. 1, pp. 15–22, 1998.
Rogozan, A., Delelise, P.: Adaptive fusion of acoustic and visual sources for automatic speech recognition, Speech Communication, vol. 26, pp. 149–161, 1998.
Tamura, S., Waibel, A.: Noise reduction using connectionist models, IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 553–556, 1988.
TIMIT: Acoustic-phonetic continuous speech corpus, Nist Speech Disc 1-1.1, October 1990.
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.: Phoneme recognition using time-delay neural networks, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp. 328–339, 1989.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lee, S., Yook, D. (2002). Viseme Recognition Experiment Using Context Dependent Hidden Markov Models. In: Yin, H., Allinson, N., Freeman, R., Keane, J., Hubbard, S. (eds) Intelligent Data Engineering and Automated Learning — IDEAL 2002. IDEAL 2002. Lecture Notes in Computer Science, vol 2412. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45675-9_84
Download citation
DOI: https://doi.org/10.1007/3-540-45675-9_84
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44025-3
Online ISBN: 978-3-540-45675-9
eBook Packages: Springer Book Archive