Abstract
Handwritten word recognition is one of the hot topics in automatic handwritten text recognition that received a lot of attention in recent years. Unlike character recognition, word recognition deals with considerable variations in word shape and written style. This paper proposes a novel deep model for language-independent handwritten word recognition. The proposed deep structure has two parallel stages for jointly learning character and word-level information. In the character-level stage, a weakly character segmentation method is performed and then applies a series of Long short-term memory (LSTM) layers for character-level representation. The word-level stage employs a series of convolutional layers for the shape and structure representation of the word. These representations are then concatenated and followed by a series of fully connected layers for jointly learning the words and the character-level information. Since the character segmentation is language independent and error-prone, the proposed deep structure only applies weakly separation scheme and does not rely on any character segmentation algorithm. Thus, it effectively utilizes character level representation without bounding on any language model. In the proposed methodology, we use two new data augmentation strategies based on a psychological assumption to increase the model generalization performance. Experimental results on five public datasets including Arabic, English and German languages demonstrate that the proposed deep model has a superior performance to the state-of-the-art methods.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ajmire P (2012) Structural features for character recognition systema review
AlKhateeb JH, Ren J, Jiang J, Ipson SS, Abed HE (2008) Word-based handwritten arabic scripts recognition using dct features and neural network classifier. In: 2008 5th international multi-conference on systems, signals and devices, pp 1–5
AlKhateeb JHY, Ren J, Jiang J, Al-Muhtaseb H (2011) Offline handwritten arabic cursive text recognition using hidden markov models and re-ranking. Pattern Recogn Lett 32:1081–1088
Almazán J, Gordo A, Fornés A, Valveny E (2014) Word spotting and recognition with embedded attributes. IEEE Transactions on Pattern Analysis and Machine Intelligence 36:2552–2566
Amrouch M, Rabi M, Es-Saady Y (2018) Convolutional feature learning and cnn based hmm for arabic handwriting recognition. In: International conference on image and signal processing. Springer, pp 265–274
Azad R, Davami F, Shayegh HR (2014) Recognition of handwritten persian/arabic numerals based on robust feature set and k-nn classifier. arXiv:1407.6492
Babu N, Soumya A (2019) Character recognition in historical handwritten documents – a survey. In: 2019 international conference on communication and signal processing (ICCSP), pp 0299– 0304
Bernardo LS, Quezada A, Munoz R, Maia FM, Pereira CR, Wu W, de Albuquerque VHC (2019) Handwritten pattern recognition for early parkinson’s disease diagnosis. http://www.sciencedirect.com/science/article/pii/S016786551930114X, vol 125, pp 78–84
Bhunia AK, Konwer A, Bhowmick A, Bhunia AK, Roy PP, Pal U (2019) Script identification in natural scene image and video frames using an attention based convolutional-lstm network. Pattern Recogn 85:172–184
Bluche T, Messina RO (2017). In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 01, pp 646–651
Bluche T, Ney H, Kermorvant C (2013) Tandem hmm with convolutional neural network for handwritten word recognition. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 2390–2394
Bluche T, Ney H, Kermorvant C (2014) A comparison of sequence-trained deep neural networks and recurrent neural networks optical modeling for handwriting recognition. In: SLSP
Boquera SE, Bleda MJC, Gorbe-Moya J, Zamora-Martínez F (2011) Improving offline handwritten text recognition with hybrid hmm/ann models. IEEE Transactions on Pattern Analysis and Machine Intelligence 33:767–779
Castro D, Bezerra BLD, Valenca M (2018) Boosting the deep multidimensional long-short-term memory network for handwritten recognition systems. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR), pp 127–132
Chen Z, Wu Y, Yin F, Liu C-L (2017) Simultaneous script identification and handwriting recognition via multi-task learning of recurrent neural networks. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 01, pp 525–530
Chowdhury A, Vig L (2018) An efficient end-to-end neural model for handwritten text recognition. In: BMVC
Dutta K, Krishnan P, Mathew M, Jawahar CV (2018) Improving cnn-rnn hybrid networks for handwriting recognition. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR), pp 80–85
de Oliveira LES, Sabourin R, Bortolozzi F, Suen CY (2002) Automatic recognition of handwritten numerical strings: a recognition and verification strategy. IEEE Trans Pattern Anal Mach Intell 24:1438–1454
Elleuch M, Maalej R, Kherallah M (2016) A new design based-svm of the cnn classifier architecture with dropout for offline arabic handwritten recognition. In: ICCS
Eraqi HM, Abdelazeem S, Rashwan MAA (2016) Combining analytical and holistic strategies for handwriting recognition. In: 2016 15th IEEE international conference on machine learning and applications (ICMLA), pp 993–997
Feng S (2019) A novel variational model for noise robust document image binarization. Neurocomputing 325:288–302
Graves A (2013) Generating sequences with recurrent neural networks. arXiv:1308.0850
Grdiet P, Garg NK (2013) Binarization techniques used for grey scale images
Hallale SB, Salunke PGD (2013) Twelve directional feature extraction for handwritten english character recognition
He S, Schomaker L (2019) Deep adaptive learning for writer identification based on single handwritten word images. Pattern Recogn 88:64–74
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780
Hu M-K (1962) Visual pattern recognition by moment invariants. IRE Trans Information Theory 8:179–187
Hussein ME, Torki M, Elsallamy A, Fayyaz M (2014) Alexu-word: a new dataset for isolated-word closed-vocabulary offline arabic handwriting recognition. arXiv:1411.4670
Ingle RR, Fujii Y, Deselaers T, Baccash J, Popat AC A scalable handwritten text recognition system. arXiv:1904.09150
Jaderberg M, Simonyan K, Vedaldi A, Zisserman A (2015) Reading text in the wild with convolutional neural networks. Int J Comput Vis 116:1–20
Jaderberg M, Vedaldi A, Zisserman A (2014) Deep features for text spotting. In: ECCV
Jayech K, Mahjoub MA, Amara NEB (2016) Arabic handwritten word recognition based on dynamic bayesian network. Int Arab J Inf Technol 13:1024–1031
Kacalak W, Stuart KD, Majewski M (2007) Selected problems of intelligent handwriting recognition. In: Analysis and design of intelligent systems using soft computing techniques
Kadhm MS, Karim A, Hassan A (2015) Handwriting word recognition based on svm classifier
Kessentini Y, Paquet T, Hamadou AB (2010) Off-line handwritten word recognition using multi-stream hidden markov models. Pattern Recogn Lett 31:60–70
Kleber F, Fiel S, Diem M, Sablatnig H (2013) Cvl-database: an off-line database for writer retrieval, writer identification and word spotting. In: 2013 12th international conference on document analysis and recognition, pp 560–564
Kozielski M, Forster J, Ney H (2012) Moment-based image normalization for handwritten text recognition. In: 2012 international conference on frontiers in handwriting recognition, pp 256–261
Lawgali A (2015) A survey on arabic character recognition
LeCun Y (1998) Gradient-based learning applied to document recognition
Maalej R, Kherallah M (2018) Convolutional neural network and blstm for offline arabic handwriting recognition. In: 2018 international arab conference on information technology (ACIT), pp 1–6
Mamathah HR, Madireddi S, Srikantamurthy KR (2012) Performance analysis of various filters for de-noising of handwritten kannada documents
Marti U-V, Bunke H (2002) The iam-database: an english sentence database for offline handwriting recognition. Int J Doc Anal Recognit 5:39–46
Mhiri M, Desrosiers C, Cheriet M (2018) Convolutional pyramid of bidirectional character sequences for the recognition of handwritten words. Pattern Recogn Lett 111:87–93
Mhiri M, Desrosiers C, Cheriet M (2019) Word spotting and recognition via a joint deep embedding of image and text. Pattern Recogn 88:312–320
Panwar S, Nain N (2012) Handwritten text documents binarization and skew normalization approaches. In: 2012 4th international conference on intelligent human computer interaction (IHCI), pp 1–6
Pechwitz M, Maddouri S, Märgner V, Ellouze N, Amiri H (2002) Ifn/enit - database of handwritten arabic words
Ptucha RW, Such FP, Pillai S, Brockler F, Singh V, Paul H (2019) Intelligent character recognition using fully convolutional neural networks. Pattern Recogn 88:604–613
Rajora S, Kumar Vishwakarma D, Singh K, Prasad M (2018) Csgi: a deep learning based approach for marijuana leaves strain classification. In: 2018 IEEE 9th annual information technology, electronics and mobile communication conference (IEMCON). IEEE, pp 209–214
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: MICCAI
Sharan A (1993) Character recognition using fourier coefficients, Ph.D. thesis
Shi B, Bai X, Yao C (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans Pattern Anal Mach Intell 39:2298–2304
Shi X, Chen Z, Wang H, Yeung D-Y, Wong W-K, chun Woo W (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting. In: NIPS
Sonkusare M, Sahu N (2016) A survey on handwritten character recognition (hcr) techniques for english alphabets. Adv Vis Comput Int J 3(1):1–12
Sudarsan D, Joseph S (2018) A novel approach for handwriting recognition in malayalam manuscripts using contour detection and convolutional neural nets. In: 2018 international conference on advances in computing, communications and informatics (ICACCI), pp 1818–1824
Sueiras J, Ruíz V, Sánchez Á, Vélez JF (2018) Offline continuous handwriting recognition using sequence to sequence neural networks. Neurocomputing 289:119–128
Sulaiman WNA, Omar MA (2019) Nasrudin, degraded historical document binarization: a review on issues, challenges, techniques and future directions
Sulaiman A, Omar K, Nasrudin MF, Arram A (2019) Length independent writer identification based on the fusion of deep and hand-crafted descriptors. IEEE Access 7:91772–91784
Tavoli R, Keyvanpour MR, Mozaffari S (2018) Statistical geometric components of straight lines (sgcsl) feature extraction method for offline arabic/persian handwritten words recognition. IET Image Process 12:1606–1616
Wang W-L, Tang MH (2015) A normalization process to standardize handwriting data collected from multiple resources for recognition. Procedia Computer Science 61:402–409. Complex Adaptive Systems San Jose, CA November 2–4, 2015
Xue H, Govindaraju V (2006) Hidden markov models combining discrete symbols and continuous attributes in handwriting recognition. IEEE Trans Pattern Anal Mach Intell 28:458–462
Yan H (1993) Skew correction of document images using interline cross-correlation. CVGIP: Graphical Model and Image Processing 55:538–543
Acknowledgements
This research was funded by the Ministry of Higher Education in Malaysia by the grant number FRGS/1/2016/ICT02/UKM/01/1.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sulaiman, A., Omar, K. & Nasrudin, M.F. Two streams deep neural network for handwriting word recognition. Multimed Tools Appl 80, 5473–5494 (2021). https://doi.org/10.1007/s11042-020-09923-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09923-1