research-article

Independent Bayesian classifier combination based sign language recognition using facial expression

Authors:

Partha Pratim Roy,

Debi Prosad DograAuthors Info & Claims

Information Sciences—Informatics and Computer Science, Intelligent Systems, Applications: An International Journal, Volume 428, Issue C

Pages 30 - 48

https://doi.org/10.1016/j.ins.2017.10.046

Published: 01 February 2018 Publication History

Abstract

Automatic Sign Language Recognition (SLR) systems are usually designed by means of recognizing hand and finger gestures. However, facial expressions play an important role to represent the emotional states during sign language communication, has not yet been analyzed to its fullest potential in SLR systems. A SLR system is incomplete without the signers facial expressions corresponding to the sign gesture. In this paper, we present a novel multimodal framework for SLR system by incorporating facial expression with sign gesture using two different sensors, namely Leap motion and Kinect. Sign gestures are recorded using Leap motion and simultaneously a Kinect is used to capture the facial data of the signer. We have collected a dataset of 51 dynamic sign word gestures. The recognition is performed using Hidden Markov Model (HMM). Next, we have applied Independent Bayesian Classification Combination (IBCC) approach to combine the decision of different modalities for improving recognition performance. Our analysis shows promising results with recognition rates of 96.05% and 94.27% for single and double hand gestures, respectively. The proposed multimodal framework achieves 1.84% and 2.60% gains as compared to uni-modal framework on single and double hand gestures, respectively.

References

[1]

F.R. Al-Osaimi, M. Bennamoun, A. Mian, Spatially optimized data-level fusion of texture and shape for face recognition, IEEE Trans. Image Process., 21 (2012) 859-872.

Digital Library

[2]

M. Al-Rousan, K. Assaleh, A. Talaa, Video-based signer-independent arabic sign language recognition using hidden Markov models, Appl. Soft. Comput, 9 (2009) 990-999.

Digital Library

[3]

S.G.M. Almeida, F.G. Guimares, J.A. Ramrez, Feature extraction in brazilian sign language recognition based on phonological structure and using RGB-D sensors, Expert Syst. Appl., 41 (2014) 7259-7271.

Digital Library

[4]

O. Aran, T. Burger, A. Caplier, L. Akarun, A belief-based sequential fusion approach for fusing manual signs and non-manual signals, Pattern Recogn., 42 (2009) 812-822.

Digital Library

[5]

B. Bauer, H. Hienz, Relevant features for video-based continuous sign language recognition, 2000.

[6]

S.K. Behera, P. Kumar, D.P. Dogra, P.P. Roy, Fast signature spotting in continuous air writing, 2017.

[7]

C. Bellmore, R. Ptucha, A. Savakis, Interactive display using depth and RGB sensors for face and gesture control, 2011.

[8]

S. Berretti, A. Del Bimbo, P. Pala, Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans, Vis. Comput., 29 (2013) 1333-1350.

Digital Library

[9]

U. Canzler, T. Dziurzyk, Extraction of non manual features for videobased sign language recognition, 2002.

[10]

G. Caridakis, S. Asteriadis, K. Karpouzis, Non-manual cues in automatic sign language recognition, Pers. Ubiquitous Comput., 18 (2014) 37-46.

Digital Library

[11]

G. Caridakis, K. Karpouzis, S. Kollias, User and context adaptive neural networks for emotion recognition, Neurocomputing, 71 (2008) 2553-2562.

Digital Library

[12]

S. Chen, Y. Tian, Q. Liu, D.N. Metaxas, Recognizing expressions from face and body gesture by temporal normalized motion and appearance features, Image Vis. Comput., 31 (2013) 175-185.

Digital Library

[13]

C.-H. Chuan, E. Regina, C. Guardino, American sign language recognition using leap motion sensor, 2014.

[14]

C.-H. Chuan, E. Regina, C. Guardino, American sign language recognition using leap motion sensor, 2014.

[15]

H. Cooper, E.-J. Ong, N. Pugeault, R. Bowden, Sign language recognition using sub-units, J. Mach. Learn. Res., 13 (2012) 2205-2231.

Digital Library

[16]

T.M.U. De Arajo, F.L. Ferreira, D.A. Silva, L.D. Oliveira, E.L. Falco, L.A. Domingues, V.F. Martins, I.A. Portela, Y.S. Nbrega, H.R. Lima, An approach to generate and embed sign language video tracks into multimedia contents, Inf. Sci. (NY), 281 (2014) 762-780.

[17]

A. Graves, M. Liwicki, S. Fernndez, R. Bertolami, H. Bunke, J. Schmidhuber, A novel connectionist system for unconstrained handwriting recognition, IEEE Trans. Pattern Anal. Mach. Intell., 31 (2009) 855-868.

Digital Library

[18]

A. Graves, J. Schmidhuber, Framewise phoneme classification with bidirectional LSTM and other neural network architectures, Neural Netw., 18 (2005) 602-610.

Digital Library

[19]

R. Hong, M. Wang, M. Xu, S. Yan, T.-S. Chua, Dynamic captioning: video accessibility enhancement for hearing impairment, 2010.

[20]

Q. Jin, P.F. Schulam, S. Rawat, S. Burger, D. Ding, F. Metze, Event-based video retrieval using audio, 2012.

[21]

P. Kumar, H. Gauba, P.P. Roy, D.P. Dogra, Coupled hmm-based multi-sensor data fusion for sign language recognition, Pattern Recogn. Lett., 86 (2017) 1-8.

Digital Library

[22]

P. Kumar, H. Gauba, P.P. Roy, D.P. Dogra, A multimodal framework for sensor based sign language recognition, Neurocomputing, 259 (2017) 21-38.

[23]

P. Kumar, R. Saini, S.K. Behera, D.P. Dogra, P.P. Roy, Real-time recognition of sign language gestures and air-writing using leap motion, 2017.

[24]

P. Kumar, R. Saini, P.P. Roy, D.P. Dogra, Study of text segmentation and recognition using leap motion sensor, IEEE Sens. J., 17 (2016) 1293-1301.

[25]

P. Kumar, R. Saini, P.P. Roy, D.P. Dogra, A position and rotation invariant framework for sign language recognition (SLR) using kinect, Multimed. Tools Appl. (2017) 1-24.

[26]

B.Y. Li, A. Mian, W. Liu, A. Krishna, Using kinect for face recognition under varying poses, expressions, illumination and disguise, 2013.

[27]

S.-Z. Li, B. Yu, W. Wu, S.-Z. Su, R.-R. Ji, Feature learning based on SAEPCA network for human gesture recognition in RGBD images, Neurocomputing, 151 (2015) 565-573.

[28]

J. Ma, W. Gao, R. Wang, A Parallel Multistream Model for Integration of Sign Language Recognition and Lip Motion, 2000.

[29]

F. Malawski, B. Kwolek, S. Sako, Using kinect for facial expression recognition under varying poses and illumination, 2014.

[30]

M. Mohandes, S. Aliyu, M. Deriche, Arabic sign language recognition using the leap motion controller, 2014.

[31]

T.D. Nguyen, S. Ranganath, Facial expressions in american sign language: tracking and recognition, Pattern Recogn., 45 (2012) 1877-1891.

Digital Library

[32]

F. Pedersoli, S. Benini, N. Adami, R. Leonardi, Xkin: an open source framework for hand pose and gesture recognition using kinect, Vis. Comput., 30 (2014) 1107-1122.

Digital Library

[33]

L.E. Potter, J. Araullo, L. Carter, The leap motion controller: a view on sign language, 2013.

[34]

L. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Proc. IEEE, 77 (1989) 257-286.

[35]

M. Reale, P. Liu, L. Yin, Using eye gaze, head pose, and facial expression for personalized non-player character interaction, 2011.

[36]

Z. Ren, J. Yuan, J. Meng, Z. Zhang, Robust part-based hand gesture recognition using kinect sensor, IEEE Trans. Multimed., 15 (2013) 1110-1120.

Digital Library

[37]

A. Savran, R. Gur, R. Verma, Automatic detection of emotion valence on faces using consumer depth cameras, 2013.

[38]

A. Shrivastava, M. Rastegari, S. Shekhar, R. Chellappa, L.S. Davis, Class consistent multi-modal fusion with binary features, 2015.

[39]

E. Simpson, S.J. Roberts, A. Smith, C. Lintott, Bayesian combination of multiple, imperfect classifiers, Neural Inf. Process. Syst. (2011).

[40]

M. Turk, A. Pentland, Eigenfaces for recognition, J. Cogn. Neurosci., 3 (1991) 71-86.

Digital Library

[41]

N. Ueda, Y. Tanaka, A. Fujino, Robust Naive Bayes combination of multiple classifications, 2014.

[42]

K. Vamsikrishna, D.P. Dogra, M.S. Desarkar, Computer vision assisted palm rehabilitation with supervised learning, IEEE Trans. Biomed. Eng., 63 (2015) 991-1001.

[43]

U. Von Agris, M. Knorr, K.-F. Kraiss, The significance of facial features for automatic sign language recognition, 2008.

[44]

H. Wang, X. Chai, X. Chen, Sparse observation (SO) alignment for sign language recognition, Neurocomputing, 175 (2016) 674-685.

Digital Library

[45]

X. Wang, M. Kankanhalli, Multifusion: a boosting approach for multimedia fusion, ACM Trans. Multimed. Comput. Commun. Appl., 6 (2010) 25.

Digital Library

[46]

F. Weichert, D. Bachmann, B. Rudak, D. Fisseler, Analysis of the accuracy and robustness of the leap motion controller, Sensors, 13 (2013) 6380-6393.

[47]

C. Xu, D. Tao, C. Xu, Multi-view intact space learning, IEEE Trans. Pattern Anal. Mach. Intell., 37 (2015) 2531-2544.

Digital Library

[48]

H.-D. Yang, S.-W. Lee, Combination of manual and non-manual features for sign language recognition based on conditional random field and active appearance model, 2011.

[49]

H.-D. Yang, S.-W. Lee, Robust sign language recognition by combining manual and non-manual features based on conditional random field and support vector machine, Pattern Recogn. Lett., 34 (2013) 2051-2056.

Digital Library

[50]

Z. Zafrulla, H. Brashear, T. Starner, H. Hamilton, P. Presti, American sign language recognition with the kinect, 2011.

Cited By

Srivastava SSingh SPooja Prakash S(2024)Continuous Sign Language Recognition System Using Deep Learning with MediaPipe HolisticWireless Personal Communications: An International Journal10.1007/s11277-024-11356-0137:3(1455-1468)Online publication date: 16-Jul-2024
https://dl.acm.org/doi/10.1007/s11277-024-11356-0
Javaid SRizvi S(2023)Manual and non-manual sign language recognition framework using hybrid deep learning techniquesJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23056045:3(3823-3833)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-230560
Moustafa AMohd Rahim MBouallegue BKhattab MSoliman ATharwat GAhmed A(2023)Integrated Mediapipe with a CNN Model for Arabic Sign Language RecognitionJournal of Electrical and Computer Engineering10.1155/2023/88707502023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/8870750
Show More Cited By

Independent Bayesian classifier combination based sign language recognition using facial expression
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning

Recommendations

Sign language recognition using a combination of new vision based features

Sign languages are based on four components hand shape, place of articulation, hand orientation, and movement. This paper presents a novel combination of vision based features in order to enhance the recognition of underlying signs. Three features are ...
A position and rotation invariant framework for sign language recognition (SLR) using Kinect

Sign language is the only means of communication for speech and hearing impaired people. Using machine translation, Sign Language Recognition (SLR) systems provide medium of communication between speech and hearing impaired and others who have ...
Towards subject independent continuous sign language recognition

This paper presents a segment-based probabilistic approach to robustly recognize continuous sign language sentences. The recognition strategy is based on a two-layer conditional random field (CRF) model, where the lower layer processes the component ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Sciences: an International Journal

Information Sciences: an International Journal Volume 428, Issue C

February 2018

136 pages

ISSN:0020-0255

Issue’s Table of Contents

Copyright © Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 February 2018

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Srivastava SSingh SPooja Prakash S(2024)Continuous Sign Language Recognition System Using Deep Learning with MediaPipe HolisticWireless Personal Communications: An International Journal10.1007/s11277-024-11356-0137:3(1455-1468)Online publication date: 16-Jul-2024
https://dl.acm.org/doi/10.1007/s11277-024-11356-0
Javaid SRizvi S(2023)Manual and non-manual sign language recognition framework using hybrid deep learning techniquesJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23056045:3(3823-3833)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.3233/JIFS-230560
Moustafa AMohd Rahim MBouallegue BKhattab MSoliman ATharwat GAhmed A(2023)Integrated Mediapipe with a CNN Model for Arabic Sign Language RecognitionJournal of Electrical and Computer Engineering10.1155/2023/88707502023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/8870750
Jin YChoi SGao YLi JLi ZJin Z(2023)TransASL: A Smart Glass based Comprehensive ASL Recognizer in Daily LifeProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584071(802-818)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584071
Sharma SSingh S(2023)A Spatio-Temporal Framework for Dynamic Indian Sign Language RecognitionWireless Personal Communications: An International Journal10.1007/s11277-023-10730-8132:4(2527-2541)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1007/s11277-023-10730-8
D AEluri S(2023)A novel Leaky Rectified Triangle Linear Unit based Deep Convolutional Neural Network for facial emotion recognitionMultimedia Tools and Applications10.1007/s11042-022-14186-z82:12(18669-18689)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1007/s11042-022-14186-z
Ma XWang QZheng TYuan L(2023)A shapelet-based framework for large-scale word-level sign language database auto-constructionNeural Computing and Applications10.1007/s00521-022-08018-235:1(253-274)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1007/s00521-022-08018-2
Sluÿters AOusmer MRoselli PVanderdonckt J(2022)QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion ControllerProceedings of the ACM on Human-Computer Interaction10.1145/35322116:EICS(1-47)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532211
Sharma SSingh S(2022)Recognition of Indian Sign Language (ISL) Using Deep Learning ModelWireless Personal Communications: An International Journal10.1007/s11277-021-09152-1123:1(671-692)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s11277-021-09152-1
Karacı AAkyol KTurut M(2021)Real-Time Turkish Sign Language Recognition Using Cascade Voting Approach with Handcrafted FeaturesApplied Computer Systems10.2478/acss-2021-000226:1(12-21)Online publication date: 1-May-2021
https://dl.acm.org/doi/10.2478/acss-2021-0002
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents