Leveraging Bidirectionl LSTM with CRFs for Pashto Tagging

POS tagging for a langauge plays an important role for various applications, such as translation, text summarization, and simplification. For Pashto language, the research work on POS tagging was first done in 2009 [Rabbi et al. 2009]. The proposed approach was based on the lexicon and rule. The rules were defined for each tag, and if two tags exist in the lexicon, then use rules to disambiguate the situation. They achieved an accuracy of 88%, but need lots of manual rules and language-specific knowledge [Rabbi et al. 2009]. An approach of projecting Farsi POS data to tag Pashto text is proposed [Khan et al. 2011]. In this research work, Farsi corpus is modified with Pashto words and lexical rules from Pashto language, this is also knows as “Pashtifications.’ Pashitification rules were used to bridge the two languages. The modified corpus was used to train a modified trigram Hidden Markov Model (HMM) model. An extended reduced tagset of Farsi and Pashto was used, accuracy of 70.84% was reported after applying morphological analysis, lexical rules, and Pashtification.

There is a lot of research work that has been done for POS tagging for other low-resource languages, like Sindhi, Bingali, Chinese, and Urdu [Sodhar et al. 2020; Khan et al. 2019]. Rule-based semantic POS tagging is used for Sindhi language as Sindhi language is highly homographic language [Mahar and Memon 2010]. Lack of diacritic in Sindhi language creates lexical and morphological ambiguity. Wordnet has been used for POS tagging system. Several algorithms have been used, including disambiguation rules, word tagging, and word tokenization for Sindhi words. An accuracy of 97% has been reported, and a morphological analyzer using the paradigm-based approach has been proposed for the Sindhi language [Motlani et al. 2016]. A morphological analyzer is also proposed for Saraiki language by Alam et al. [2023] for the first time. It provides 83% coverage, 93% precision, and 96% recall. POS tagging for Urdu language has been proposed by various studies. Nasim et al. [2020] used BiLSTM CRF model for POS tagging for Urdu language. The accuracy (96% F1 score) indicates that the model outperforms existing approaches for Urdu language POS tagging. Different Machine Learning techniques have been applied on Urdu POS tagging by different researchers [Laeeq et al. 2022; Tehseen et al. 2023].

Rule-based approach is the earliest approach for POS tagging proposed by Klein and Simmons [1963]. In rule-based approach, manually hand-written grammar rules are used with some annotated data stored in corpus. Rules-based systems are accurate and fast, but their drawback is that they require a number of manually hand-written rules [Dwivedi and Malakar 2015]. An example of a rule is, “If an ambiguous/unknown word X is preceded by a Determiner and followed by a Noun, tag it as an Adjective” [Antony and Soman 2011]. Rule-based systems are further divided into lexical rules and contextual rules. Lexical rules find the label of a word based on lexical properties while contextual rules use context to find the label for a word given a context. Accuracy of Rule-based Approach per word was 96% [Brill 1992], later Brill improved accuracy of these approaches to 97.5% by using transformation-based error driven learning [Brill 1995]. Transformation-based approach not only considers the previous word but also considers the next word.

In statistical-based approaches, frequencies and probabilities of words are used. The training process of such models is relatively faster as compared to other models. Disadvantages of such approaches is that sometimes they assign tag sequence to sentences that are not correct as per grammar of the language. CLAWS tagger is a tagger uses a Statistical-based approach (Hidden Markov Model) and achieves an accuracy of 97% [Garside 1987]. Banko and Moore modified HMM and proposed a contextual model that can consider both previous and next word to extend the context. They achieved accuracy of 96.59% [Banko and Moore 2004]. Stanford NLP community also worked on a wide range of POS tagger. They worked on Bayesian Approach and used a structure similar to HMM. They reported accuracy near to state of the art taggers [Goldwater and Griffiths 2007]. Stanford NLP group worked on Unsupervised Dependency Parsing without Gold Part-of-Speech Tags [Goldwater and Griffiths 2007]. They explored Unsupervised methods for POS tagging. Also, they explored and removed limitation of Gold work that a word will always have the same tag. They further proposed a method that successfully assigns different tags to the same words in different context. They reported an accuracy of 59.1% directed accuracy on Section 3 all sentences of Wall Street Journal benchmark corpus [Spitkovsky et al. 2011; Charniak et al. 2000].

Maximum Entropy-based approach POS tagging is performed by assigning the tag sequence to words sequence having maximum likelihood. Information from diverse sources can also be incorporated [Toutanova and Manning 2000]. In this approach, we model the known and ignore the unknown [Berger et al. 1996]. Ratnaparkhi, proposed Maximum Entropy-based Approach to POS tagging for English. In his study, he explored that Maximum Entropy-based Approach is flexible for virtually unrestricted feature set. The overall accuracy of this approach is same as that of state-of-the-art POS taggers. The sentence level accuracy achieved was 47.55% with dictionary used and 47.38% without dictionary [Ratnaparkhi 1996]. Toutanova also Explored Maximum Entropy-based Approach for POS tagging of English language. In his work, he incorporated extra information of (case information, form of verbs, and features for distinguishing particles from preposition and adverb) with his model. According to the findings, the feature of using prefixes impacts negatively on accuracy, therefore feature of prefixes was removed, which leads to the improvement in accuracy of unknown words as well as the overall accuracy. The overall accuracy for this work was reported as 96.86% for seen and 86.91% for previously unseen words [Toutanova and Manning 2000]. Ekbal used Maximum Entropy-based Approach to POS tagging for Bengali language. In this study, Named Entity Tag information was used with Maximum Entropy-based approach. It was further explored that at the time of testing for unknown words use of lexicon and Named Entity tag gives more accurate results. The overall accuracy achieved was 88.2%, which is an 8% improvement over HMM-based tagger for Bengali language [Ekbal et al. 2008].

Using this approach, the task of POS tagging is viewed as sequence classification task. The input word with its neighboring words and the associated tag is given to network, the network train over the training examples and adjust its parameters. the trained network is further used for inference.

In 2011, Collobert Proposed a new Approach for various NLP tasks including POS tagging, chunking, named entity recognition, and semantic role labeling. It was explored that use of hand-crafted features can be replaced by applying deep layers of neural network. Per-word accuracy reported for POS tagging was 96.37, F1 score of 90.33 for chunking, F1 score of 81.47 for NER, and 70.99 for SRL was reported [Collobert et al. 2011]. Zheng and Chen proposed a Deep Learning-based Approach to POS tagging and Chinese word Segmentation. The approach mainly focused on avoiding task specific feature engineering by applying Deep Learning. The accuracy reported for joint task of word segmentation and POS tagging was 91.82 F-scores. The accuracy achieved by this approach is near to state of the art with the cost of avoiding task-specific feature engineering [Zheng et al. 2013]. Santos and Zadrozny proposed Deep Learning-based Approach to POS tagging of English and Portuguese language. In this study, they used character level representation for words, to avoid hand craft features, a technique known as word embedding at character level was used. The main finding of this study was incorporating morphological knowledge to the model without any hand craft features. The accuracy achieved was 97.32% and 97.47% for English and Portuguese language, respectively [Santos and Zadrozny 2014]. Wang proposed an approach of Deep Learning for POS tagging, he explored new way of tagging sequential data. He used Bidirectional Long Short-term Memory–Recurrent Neural Network (BLSTM-RNN). Main findings of this research work are effective use of BLSTM-RNN to POS tagging without using any morphological knowledge with state of the art accuracy, and a new approach to word Embedding. The accuracy reported was 97.40% [Wang et al. 2015]. Ma and Hovy proposed an approach using deep learning called Bidirectional Long Short-term Memory–Convolution Neural Network (BLSTM-CNN) for POS tagging and Named Entity Recognition (NER). In this research work, the use of character level representation was also explored with word-level representation. Accuracy reported for POS tagging was 97.55% and 91.21 F1 score for NER [Ma and Hovy 2016]. Plank and his colleague also explored application of BLSTM for POS tagging on 22 different languages, they applied BLSTM with different input representation, word-level embedding, character level embedding, and Unicode byte level embedding, they evaluated that BLSTM works at the performance of state of the art for 22 different languages, specially works well with morphologically rich languages. It was further explored that performance of BLSTM is less dependent on size of training data, and label corruption with little noise [Plank et al. 2016].

In hybrid approaches, Rule-based approach is either combined with Statistical-based approach or with Neural Network-based approach. In such approaches, firstly statistical approach is applied and then rules are used for corrections. Lee proposed a hybrid method for POS tagging of Korean text. Lee combined HMM- and rule-based approaches and a morphological analyzer. Lee achieved an overall accuracy of 94.9% [Lee et al. 1997]. Murata used a hybrid approach for Thai language [Ma et al. 2000]. In his work, he combined neural network-based approach with Rule-based approach. Murata concluded that the hybrid approach performs better than HMM. Hadni proposed a hybrid method by combining rule-based approach with HMM-based approach for Arabic language. Hadni used four POS tag classes—Noun, Verb, Particle, and Quranic Initial. Accuracy achieved by this work is 94.4% for Rule-based and 97.6% after HMM is applied [Hadni et al. n.d.]. Sinha proposed a hybrid method by combining corpus-based approach with rules-based and HMM. In this work, first input word is checked in lexicon, if found then tag is assigned to it, otherwise the word is passed to HMM module and most likely tag is assigned. Next the word-tag sequence is passed to rule-based module for correction or making new entry to lexicon in case of unknown word [Sinha et al. 2015].

Hidden Markov Model is a probabilistic and statistical model that is based on a property called Markov property, Markov property states that the future is only dependent on the present, not the past. Observing at a particular state of the model, we can only think of the likelihood of next state to be occurred (transition or jump forward). Also at a particular state, we can think of events that occur at that state. Hidden Markov Model uses transition probabilities \(P_{T}\) for considering the co-occurrence of words and capturing context. Also, emission probabilities \(P_{E}\) for considering the chance of occurring an event given the present state [Schönhuth 2009]. For POS tagging problem consider words of a sentence as \((w_1 \ldots w_n)\) and tagset as \((t_1 \ldots t_n)\) more formally, we can define a Markov model for POS tagging as follows [Charniak et al. 1993]:

In Equation (1) first clause of product \(P(t_i | t_{i-1})\) is the probability of tag \(t_i\) given tag \(t_{i-1}\) also know as transition probabilities and the second clause \(P(w_i| t_i)\) is the probability of word \(w_i\) given the tag \(t_i\) also know as emission probabilities.

HMM is a well-known model for tagging sequential data such as Part-of-Speech tagging. Various HMM-based approaches have been proposed for POS tagging, but there is no such approach for Pahsto language that exploits HMM for POS tagging. In this work, we tailored [Rudd 2009] implementation for Pashto.

There is no benchmark dataset available for the task of POS tagging for Pashto language. We created a dataset for the task of Pashto POS tagging with focus on words having more than one tag. We developed a dataset for this task by collecting sentences from native Pashto speakers, and Pashto web sites, and got these sentences checked from experts of the language. Our dataset consists of 16 ambiguous words. For every ambiguous word, we collected 20 sentences, 10 sentences for each sense.² For example, in Figure 2 the word has two senses, one for noun and other for verb. Finally, all the sentences are manually hand tagged from two online dictionaries “The Pashto online dictionary” [Khan 2017], and “Daryab Pashto dictionary” [Achakzai 2017]. The entire dataset consists of 320 sentences. These sentences form the training input for our model as shown in Figure 5.

Encoding is used in text to let the computer interpret the alphabets, words and characters of a language. Different encoding schemes are used, e.g., UTF/UNICODE, ISO-8859-x series, ASCII, Windows-1252 [Ishida 2015].

As Pashto language is not fully supported by UTF encoding, also not every Application and IDE supports Pashto language therefore manual encoding is needed.

For our system, we encoded Pashto words into integers by creating a vocabulary of unique words. We created sentences of integers by searching words of input sentence in the vocabulary and picking their indices. For making all sentences to have equal length, we padded each sentence with zeros. Example of Encoding sentences to integers is given in Figure 6. In Figure 6, there are three sentences of Pashto language. Each sentence has various number of words containing 16 unique words. Each unique word is assigned a unique number. Based on these unique numbers, three sentences are encoded.

Our proposed LSTM model consists of three layers, an Embedding layer, a bidirectional LSTM layer, and a linear layer. The embedding layer input feature size is 580 and the output feature size is 300, which means that we encode each input token into a 300-dimensional vector also known as an embedding vector. The embedding vector is passed into the LSTM layer with input features as 300 and output features as 150. Finally, the 150-dimensional latent vector is then passed into a linear layer with input features of 300 and output features size of 16. This means that we have a total of 16 tags in our tag set including the start and stop sequence.

We trained our model for 20 epochs using the train set and then reported the accuracy score using the test set. To get better performance, we performed multiple experiments with varying epochs and learning rate. We first tuned learning rate by trying different values ranging from 0.02 to 0.1 with a step of 0.01, then we tried number of epochs ranging from 200 to 1,900 with a step of 100, and we recorded accuracy each time. Figure 7 shows the plot for the training loss against number of epochs.

Encoded input sentences are converted into a format that LSTM understands. LSTM understands data in contextual format. Figure 6 represents an example of converting simple sequential data to contextual data, which is a required data format by the supervised learning model. There are three textual sentences in Figure 6. To convert them into numeric format, a vocabulary vector consisting of all the words occurring in the three sentences is built where each word is assigned a unique numeric value. on the basis of that vector, sentences are represented in numeric format. In Figure 8, we have three sentences in Encoded format. Next, they are broken down into slices by taking previous words plus one following word at a time. In this way one sentence is broken down into four sub slices. Context is handled by using parenthesis, inside parenthesis are the context words and outside the parenthesis are the focus word, the word under observation. For example [(context words), \(w_i\)], we take context length up to eight words. In the next step, embedded vectors of contextual data is fed into LSTM network in one-word vector per cell fashion. Example of a sentence fed into LSTM is given in Figure 9. LSTM updates its weights accordingly and keeps information of the context till the end of the sentence, no matter how long the sentence is. After end of the sentence, LSTM, clears its context and gradients, because we take context only at sentence level. The same process is repeated for each sentence. During each iteration, weights are adjusted so that the distance between the target and observed values is minimized. After particular number of iterations, we achieve a learned network. The learned network is then used for testing/predicting unseen sentences.

Word vector represents the encoded form of Pashto sentences. To predict an appropriate tag for each word vector, word vector is passed to LSTM. LSTM returns a vector of weights for each word of a sentence. We assign the tag that has maximum weight in the weight vector for that word. We get words and its tag in the form of integers, and then we decode it back to Pashto words/tags pairs using the dictionary we created for this purpose. After achieving predicted tags for word vectors, we calculate accuracy by comparing the predicted tags with actual tags of testing Examples. Table 1 represents an illustration of decoding scores vectors where words are assigned tags in form of word-tag pair, such as tag T5 is assigned to word W1, tag T2 is assigned to word W2, tag T10 is assigned to W3, and tag T3 is assigned to word W4.

To speed up computations and improve performance a technique is used called “dimensionality reduction.” There are two ways for dimensionality reduction, feature extraction and feature selection. Feature extraction is the process of transforming high-dimensional feature vector to low-dimensional feature vector by finding association between features whereas feature selection is the process of selecting a subset of features that best represents the class in the process of classification.

We used feature extraction to find the association between words; for feature extraction, we used Word Embedding. Word embedding can be achieved as a standalone task or can be jointly used with a model of neural network. We also used the Word Embedding jointly with our model, which takes input sentence as a vector of integers, and then transforms it to low-dimensional vector. Then the vectors are passed to hidden layers of the neural network model.

For splitting our data into training and testing, we choose two sentences for each ambiguous word, one sentence that contains ambiguous word with one kind of tag and one sentence for other kind of tag, e.g., one sentence for noun and one for verb. We separated 32 sentences, which sum up to 352 words as testing set, and the remaining 288 sentences, which sums up to 3,168 words, are used as training set.

In addition to splitting of data as described in previous section, we applied K-Fold Cross validation to our data with \(k=10\). In this we created 10 slices of data and iterated through it; in each iteration, we pick one slice for testing and remaining 9 slices for training.

We also applied Conditional Random Field BLSTM for POS tagging of Pashto. We trained/tested the system with the same dataset used for LSTM and BLSTM. We keep context size of eight words. For CRF-BLSTM, we also calculated separate accuracy for bi-tag words as well as for overall words. The results we obtained by applying CRF-BLSTM are given in Table 2.

We also applied HMM, a statistical model that works on frequencies/probabilities of words to POS tagging of Pashto language. We trained/tested HMM on the same dataset that was used for LSTM, BLSTM, and CRF-BLSTM. The context size of three words was used, because HMM is a trigram tagger; if we increase context size, then the computation cost increases exponentially. For HMM, we also measured accuracy separately, for overall words as well for bi-tag words. Table 3 shows the results we obtained after applying HMM.

Figure 10 shows nine examples of sentences with ambiguous words (highlighted in purple color). For instance, consider sentence 1, the highlighted word is a noun and the model predicted it as a noun, whereas, in sentence 2, the same word is used as a verb and the model also predicted it as a verb. Similar situationa can be observed in sentencea 3 and 4. The highlighted word is used as an adjective in sentence 3, while in sentence 4 it is used as a noun. Our model predicted it correctly in both places. This shows that our model is trained well on the ambiguous words and their tags and predicted it correctly.

We applied CRF-BLSTM and Statistical-based HMM. We conclude that, for Pashto POS tagging, CRF-BLSTM-based approaches perform better than statistical-based approaches HMM (as shown in Table 4). The key fact behind the deep learning-based approaches performance is that they keep contextual information long through the sequence. Deep learning-based approaches keep information remembered in the sequence until explicitly directed to forget it, whereas HMM keep context up to limited length (three words). Moreover, for bi-tag words HMM fails to decide which tag to assign, if each possible tag appears with equal probability in the training set.

Our current approach is supervised learning approach that relies on annotated data. The performance of supervised learning models heavily relies on the amount of training dataset. For this purpose, we need a large amount of annotated data for a low-resource language (Pashto) but the annotated data for Pashto language is not available. this is a limitation of our study.

For the task of POS tagging, there is no corpus readily available for Pashto language, which contains ambiguous words and their associated tags. Ambiguous words in Pashto may have more than one tag in different situations. For better training of the model for ambiguous words require a large number of such examples, but the unavailability of large datasets for Pashto language, it would not be possible.