research-article

An optimized machine translation technique for multi-lingual speech to sign language notation

Authors:

Amandeep Singh Dhanjal,

Williamjeet SinghAuthors Info & Claims

Multimedia Tools and Applications, Volume 81, Issue 17

Pages 24099 - 24117

https://doi.org/10.1007/s11042-022-12763-w

Published: 01 July 2022 Publication History

Abstract

Due to the lack of assistive resources, hard-of-hearing people cannot live independently. Sign language or gesture language is the natural language and it is the primary mode of communication for hard-of-hearing people. Researchers and IT companies are continuously trying to find the best solutions to minimize the communication barriers for hearing-impaired people. Existing translation techniques for speech to sign language on the web platform are consuming higher resources. This study presents an optimized technique for direct machine translation of multi-lingual speech to Indian sign language using the HamNoSys notation system, whereas existing techniques were translating speech-text-HamNoSys. Performance comparison of both existing and the proposed techniques is analyzed in this study. The proposed technique optimizes the resources for the following parameters: CPU, heap memory, primary memory, and classes load. The result shows that the existing technique takes 220 MB heap memory, 10 threads, 2236 classes, and CPU for 12 s. The proposed technique consumes only 210.4 MB, 9 threads, 2113 classes, and CPU for 9 s.

References

[1]

Abushariah MAM, Ainon RN, Zainuddin R et al (2010) Natural speaker-independent Arabic speech recognition system based on hidden Markov models using sphinx tools. In: international conference on computer and communication engineering (ICCCE’10). IEEE, 1–6.

[2]

Almasoud AM and Al-Khalifa HS SemSignWriting: a proposed semantic system for Arabic text-to-SignWriting translation J Softw Eng Appl 2012 05 08 604-612

[3]

Anuja K, Suryapriya S, Idicula SM (2009) Design and development of a frame based MT system for English-to-ISL. In: 2009 world congress on Nature & Biologically Inspired Computing (NaBIC). IEEE, 1382–1387.

[4]

Bouzid Y, Jemni M (2014) TuniSigner: a virtual interpreter to learn sign writing. In: 14th international conference on advanced learning technologies. IEEE, 601–605.

[5]

Bouzid Y, Jemni M (2013) An avatar based approach for automatically interpreting a sign language notation. Proc - 2013 IEEE 13th Int Conf Adv learn Technol ICALT 2013 92–94.

[6]

Bouzid Y, Jemni M (2013) An animated avatar to interpret signwriting transcription. International Conference on Electrical Engineering and Software Applications. IEEE, In, pp 1–5

[7]

Brour M and Benabbou A 2019 ATLASLang MTS 1: Arabic text language into Arabic sign language machine translation system. Procedia Comput Sci 148(Icds 2018): 236–245.

[8]

Buttussi F, Chittaro L, Coppo M (2007) Using Web3D technologies for visualization and search of signs in an international sign language dictionary. In: proceedings of the twelfth international conference on 3D web technology - Web3D ‘07. ACM press, New York, New York, USA, 61.

[9]

Carlos da Rocha Costa A and Pereira Dimuro G (2002) SignWriting-Based Sign Language Processing. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer, Berlin, Heidelberg, 202–205

[10]

Cox S, Lincoln M, Tryggvason J, et al. The development and evaluation of a speech-to-sign translation system to assist transactions Int J Hum Comput Interact 2003 16 2 141-161

[11]

Cox S, Lincoln M, Tryggvason J et al (2002) TESSA, a system to aid communication with deaf people. In: the fifth international ACM conference on assistive technologies. ACM. 205–212.

[12]

Dhake D, Kamble MP, Kumbhar SS, and Patil SM Sign language communication with dumb and deaf people Int J Eng Appl Sci Technol 2020 5 4 254-258

[13]

Efthimiou E, Fotinea SE, Vogler C et al (2009) Sign language recognition, generation, and modelling: a research effort with applications in deaf communication. Lect notes Comput Sci (including Subser Lect notes Artif Intell Lect notes bioinformatics) 5614 LNCS(PART 1): 21–30.

[14]

Elliott R, Glauert JRW, Kennaway JR, et al. Linguistic modelling and language-processing technologies for avatar-based sign language presentation Univers Access Inf Soc 2008 6 4 375-391

[15]

Elliott R, Glauert JRW, Kennaway JR, Marshall I (2000) The development of language processing support for the ViSiCAST project. In: proceedings of the fourth international ACM conference on assistive technologies - assets ‘00. ACM, 101–108.

[16]

Hanke T HamNoSys – representing sign language data in language resources and language processing contexts 2004 In LREC 1-6

[17]

Huenerfauth M, Zhao L, Gu E, and Allbeck J Evaluation of American sign language generation by native ASL signers ACM Trans Access Comput 2008 1 1 1-27

[18]

Su H-Y and Wu C-H Improving structural statistical machine translation for sign language with small corpus using thematic role templates as translation memory IEEE Trans Audio Speech Lang Processing 2009 17 7 1305-1315

[19]

Joy J, Balakrishnan K, and Madhavankutty S Developing a bilingual mobile dictionary for Indian sign language and gathering users experience with SignDict Assist Technol 2018 32 3 153-160

[20]

Kamal Z, Hassani H (2020) Towards Kurdish text to sign translation. Language Resources and Evaluation Conference, In, pp 117–122

[21]

Karpouzis K, Caridakis G, Fotinea SE, and Efthimiou E Educational resources and implementation of a Greek sign language synthesis architecture Comput Educ 2007 49 1 54-74

[22]

Karpov A, Kipyatkova I, Zelezny M (2016) Automatic Technologies for Processing Spoken Sign Languages. Procedia Computer Science. Elsevier, In, pp 201–207

[23]

Kaur A, Singh A (2016) Power-normalized cepstral coefficients (PNCC) for Punjabi automatic speech recognition using phone based modelling in HTK. In: 2016 2nd international conference on applied and theoretical computing and communication technology (iCATccT). IEEE, 372–375.

[24]

Kaur K, Kumar P (2016) HamNoSys to SiGML conversion system for sign language automation. Procedia Computer Science. Elsevier, In, pp 794–803

[25]

Kaur R, Singh W (2018) Speech based retrieval system for Punjabi language. In: 2018 international conference on smart systems and inventive technology (ICSSIT). IEEE, 498–502.

[26]

Kose H, Yorganci R, Algan EH, and Syrdal DS Evaluation of the robot assisted sign language tutoring using video-based studies Int J Soc Robot 2012 4 3 273-283

[27]

Kouremenos D, Fotinea SE, Efthimiou E, and Ntalianis K A prototype Greek text to Greek sign language conversion system Behav Inf Technol 2010 29 5 467-481

[28]

Krishnamoorthy P and Prasanna M Processing Noisy speech for enhancement IETE J Educ 2007 24 5 351-357

[29]

Kumar K, Aggarwal RK, and Jain A A Hindi speech recognition system for connected words using HTK Int J Comput Syst Eng 2012 1 1 25-32

[30]

Kumar Y and Singh N An automatic speech recognition system for spontaneous Punjabi speech corpus Int J Speech Technol 2017 20 2 297-303

[31]

López-Colino F, Colás J, and Garrido J Full skeleton-based virtual human animation: an improved design for SL avatars IETE Tech Rev 2017 34 1 11-21

[32]

López-Ludeña V, San-Segundo R, González C et al (2011) Methodology for developing a speech into sign language translation system in a new semantic domain. 1–11.

[33]

Luqman H and Mahmoud SA Automatic translation of Arabic text-to-Arabic sign language Univers Access Inf Soc 2019 18 4 939-951

[34]

Masbate AMM, Campus M, and St M Text to English and text to Filipino sign language translator : an android-based Mobile application with text analysis using Knuth-Morris-Pratt naive algorithm Elem Educ Online 2020 19 4 78-84

[35]

Matsumoto T, Kato M, Ikeda T (2009) JSPad - a sign language writing tool using SignWriting. In: proceedings of the 3rd international universal communication symposium. 363–367.

[36]

Morgan G Transcription of child sign language Deaf Educ Int 2003 5 3 157-166

[37]

Naert L, Larboulette C, and Gibet S Motion synthesis and editing for the generation of new sign language content Mach Transl 2021 35 405-430

[38]

Nair MS, Nimitha AP, Idicula SM (2016) Conversion of Malayalam text to Indian sign language using synthetic animation. In: 2016 international conference on next generation intelligent systems, ICNGIS 2016. IEEE, 1–4.

[39]

Nawshin S, Saif N, Mohammad AS, Jameel M (2020) Protik: Bangla sign language teaching aid for children with impaired hearing. In: 2020 IEEE region 10 symposium (TENSYMP). IEEE, 440–443.

[40]

Neves C, Coheur L, Nicolau H (2020) HamNoSyS2SiGML: translating HamNoSys into SiGML. In: LREC 2020 - 12th international conference on language resources and evaluation, conference proceedings. 6035–6039.

[41]

Oh J, Jeon S, Kim B, et al. Avatar-based sign language interpretations for weather forecast and other TV programs SMPTE Motion Imaging J 2017 126 1 57-62

[42]

Qaisar SM Isolated speech recognition and its transformation in visual signs J Electr Eng Technol 2019 14 2 955-964

[43]

Rajaganapathy S, Aravind B, Keerthana B, and Sivagami M Conversation of sign language to speech with human gestures Procedia Comput Sci 2015 50 5010-5015

[44]

Ravanelli M, Parcollet T and Bengio Y 2019 The Pytorch-kaldi Speech Recognition Toolkit. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6465–6469

[45]

Reddy CKA, Hao Y, Panahi I (2016) Two microphones spectral-coherence based speech enhancement for hearing aids using smartphone as an assistive device. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, 3670–3673.

[46]

San-Segundo R, Barra R, Córdoba R, et al. Speech to sign language translation system for Spanish Speech Commun 2008 50 11–12 1009-1020

[47]

Shahriar R, Zaman AGM, Ahmed T et al (2017) A communication platform between bangla and sign language. In: 5th IEEE region 10 humanitarian technology conference 2017, R10-HTC 2017. 1–4.

[48]

Sindhu DV, Sagar BM (2016) Study on machine translation approaches for Indian languages and their challenges. In: 2016 international conference on electrical, electronics, communication, computer and optimization techniques (ICEECCOT). IEEE, 262–267.

[49]

Stokoe WC and Marschark M Sign language structure: an outline of the visual communication systems of the american deaf J Deaf Stud Deaf Educ 2005 10 1 3-37

[50]

Stoll S, Camgoz NC, Hadfield S, and Bowden R Text2Sign: towards sign language production using neural machine translation and generative adversarial networks Int J Comput Vis 2020 128 4 891-908

[51]

Veale T, Conway A, and Collins B The challenges of cross-modal translation: English-to-sign-language translation in the Zardoz system Mach Transl 1998 13 1 81-106

[52]

Vichyaloetsiri T, Wuttidittachotti P (2017) Web service framework to translate text into sign language. In: international conference on computer, information and telecommunication systems (CITS). IEEE, Dalian, China, 180–184.

[53]

Wu C-H, Su H-Y, Chiu Y-H, Lin C-H (2007) Transfer-based statistical translation of Taiwanese sign language using PCFG. ACM trans Asian Lang Inf Process 6(1): 1-es.

[54]

Zelinka J and Kanis J 2020 Neural Sign Language Synthesis: Words Are Our Glosses. In: 2020 IEEE winter conference on applications of computer vision (WACV). IEEE, 3384–3392

[55]

Zhao L, Kipper K, Schuler W et al (2000) A machine translation system from English to American sign. Language 54–67

Cited By

Zhang LYin HLi WYin XWang Z(2023)Studying the Intelligent Translation System for English Using 3D Visualisation TechnologiesProceedings of the 3rd International Conference on Electronic Information Technology and Smart Agriculture10.1145/3641343.3641382(221-227)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3641343.3641382

Recommendations

An automatic machine translation system for multi-lingual speech to Indian sign language
Abstract
Sign language (SL) is the best suited communication medium for hearing impaired people. Even with the advancement of technology, there is a communication gap between the hearing impaired and hearing people. The aim of this research work is to ...
Text to Sign Language Translation System: A Review of Literature

Many machine translation systems for spoken languages are available, but the translation system between the spoken and Sign Language are limited. The translation from Text to Sign Language is different from the translation between spoken languages ...
Spoken Language to Sign Language Translation System Based on HamNoSys
SSPS '19: Proceedings of the 2019 International Symposium on Signal Processing Systems

Sign languages use visual-manual modality to convey information, thereby enabling communication between hearing-impaired people. However, there is no fully developed technology that enables the communication between hearing-impaired people and those ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 81, Issue 17

Jul 2022

1429 pages

ISSN:1380-7501

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2022

Accepted: 21 February 2022

Revision received: 22 October 2021

Received: 29 May 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Science for Equity, Empowerment and Development Division

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang LYin HLi WYin XWang Z(2023)Studying the Intelligent Translation System for English Using 3D Visualisation TechnologiesProceedings of the 3rd International Conference on Electronic Information Technology and Smart Agriculture10.1145/3641343.3641382(221-227)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3641343.3641382

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents