Event classification from the Urdu language text on social media

Introduction

In the current digital and innovative era, text is still the strongest and dominant source of communication instead of pictures, emoji, sounds, and animations (Lenhart et al., 2010). The innovative environment of communication; real-time availability (Motoyama et al., 2010) of the Internet and unrestricted access for communication on social networks have attracted billions of people around the world. Now, people are hooked together via the Internet like a global village. They preferred to share detailed worthy information about different topics, opinions, views, ideas, and events (Reuter & Cimiano, 2012) on social networks in different languages. The usage of different languages is being popular because social media and news channels have created space for local languages (Rogstadius et al., 2013). The Google input tool (https://www.google.com/inputtools/) provides language transliteration support for more than 88 different languages. Many other tools like the software (Inpage and Pak-Urdu for the Urdu language) provide the support to use local languages on social media for communication. Google Translate (https://translate.google.com/?hl=en) is a platform that facilitates multilingual users of more than 100 languages for conversation. Generally, people prefer to communicate in local languages instead of non-local languages for easiness.

A cursive language Urdu is one of the local languages that is being highly adapted for communication. There are more than 300 million (Livingston, 2005) Urdu language users all around the world that can speak, understand and write in the Urdu language. The Urdu language is a mix-composition of different languages i.e., Arabic, Persian, Turkish, and Hindi (Ghulam & Soomro, 2018). In Pakistan and India, more than 65 million people can speak, understand, and write the Urdu language (Naz & Hussain, 2013). It is one of the resource-poor, neglected languages (Mukund, Srihari & Peterson, 2010) and the national language (Nadeau & Sekine, 2007) of Pakistan: the 6th most populous (https://www.worldometers.info/world-population/population-by-country/) country in the world. Urdu is also widely adopted and spoke as a second language all over the Pakistan (Ghulam & Soomro, 2018; Naz & Hussain, 2013; Mukund, Srihari & Peterson, 2010; Nadeau & Sekine, 2007).

In South Asia other countries (Riaz, 2008) i.e., Bangladesh, Iran, and Afghanistan also have a considerable number of Urdu language users. Pak Urdu Installer (http://www.mbilalm.com/download/pak-urdu-installer.php) and Inpage are also common software, it support the Urdu language for textual writing (communication).

In contrast to cursive languages, there exists noteworthy work of information extraction and classification for i.e., English, French, German, and many other non-cursive languages (Nadeau & Sekine, 2007; Riaz, 2008).

Sifting worthy insights from an immense amount of heterogeneous text existing on social media is an interesting and challenging task of Natural Language Processing (NLP). Event extraction and classification is one of the NLP tasks. The information of event classification is helpful to develop various NLP applications i.e., to respond to emergencies, outbreaks, rain, flood, and earthquake (Yin et al., 2012), etc. Generally, people share their intent, appreciation, or criticism (Purohit et al., 2015) i.e., enjoying discount offers by selling brands or criticizing the quality of the product. Earlier awareness of sentimental insights can be helpful to protect from business losses. The implementation of smart-cities possess a lot of challenges; decision making, event management, communication, and information retrieval. Extracting useful insights from an immense amount of text, dramatically enhance the worth of smart cities (Alkhatibl, El Barachi & Shaalan, 2018). Event information can be used to predict the effects of the event on the community, improve security and rescue the people.

Furthermore, classification of events can be used to collect relevant information about a specific topic, top-trends, stories, text summarization, and question and answering systems (Khan et al., 2016; Jacobs, 1963). Such information can be used to predict upcoming events, situations, and happening. For example, protesting events reported on social media generally end with conflict among different parties, injuries, death of people, and misuse of resources that cause anarchy. Some proactive measurements can be taken by the state to diffuse the situation and to prevent conflict. Similarly, event classification is crucial to monitor the law-and-order situation of the world.

Extracting and classification of event information from Urdu language text is a unique, interesting, and challenging task. The features of the Urdu language that made the event classification tasks more complex and challenging are listed below.

• Cursive nature of the script

• Morphologically enriched

• Different structures of grammar

• Right to the left writing style

• No text capitalization

Similarly, the lack of resources i.e., the part-of-speech tagger (PoS), words stemmer, datasets, and word annotators are some other factors that made the processing of the Urdu text complex. There exist few noteworthy works related to the Urdu language text processing (see the literature for more details). All the above-mentioned factors motivated us to explore Urdu language text for our task.

Concept of our system

The overall working process of our proposed framework is given in Fig. 1.

Our contribution

• In this research article, we claim that we are the first ones who are exploring the Urdu language text to perform multi-class event classification at the sentence level using a machine learning approach,

• A dataset that is larger than state-of-art used in experiments. In our best knowledge classification for twelve (12) different types of events never performed,

• A comprehensive and detailed comparison of six machine learning algorithms is presented to find a more accurate model for event classification for the Urdu language text.

Our limitations

• There is no specific Word2Vec model for Urdu language text,

• There is also no availability of the free (open source) part-of-speech tagger and word stemmer for Urdu language text,

• Also, there exists no publicly available dataset of Urdu language text for sentence classification.

Related work

Classification of events from the textual dataset is a very challenging and interesting task of Natural Language Processing (NLP). An intent mining system was developed (Purohit et al., 2015) to facilitate citizens and cooperative authorities using a bag of token model. The researchers explored the hybrid feature representation for binary classification and multi-label classification. It showed a 6–7% improvement in the top-down feature set processing approach. Intelligence information retrieval plays a vital role in the management of smart cities. Such information helps to enhance security and emergency management capabilities in smart cities (Alkhatibl, El Barachi & Shaalan, 2018). The textual content on social media is explored in different ways to extract event information. Generally, the event has been defined as a verb, noun, and adjective (Nadeau & Sekine, 2007). Event detection is a generic term that is further divided into event extraction and event classification. A combined neural network of the convolutional and recurrent network was designed to extract events from English, Tamil, and Hindi languages. It showed 39.91%, 37.42% and 39.71% F_ Measure (Ahmed et al., 2016).

In the past, the researchers were impassive in cursive language, therefore a very limited amount of research work exists in cursive language i.e., Arabic, Persian Hindi, and Urdu (Alsaedi & Burnap, 2015). Similarly, in the work of (Alsaedi & Burnap, 2015), the authors developed a multiple minimal reduct extraction algorithm which is an improved version of the Quick reduct algorithm (Al-Radaideh & Al-Abrat, 2019). The purpose of developing the algorithm is to produce a set of rules that assist in the classification of Urdu sentences. For evaluation purposes, an Arabic-based corpus containing more than 2,500 documents was plugged in for classifying them into one of the nine classes. In the experiment, we compared the results of the proposed approach when using multiple and single minimal reducts. The results showed that the proposed approach had achieved an accuracy of 94% when using multiple reducts, which outperformed the single reduct method which achieved an accuracy of 86%. The results of the experiments also showed that the proposed approach outperforms both the K-NN and J48 algorithms regarding classification accuracy using the dataset on hand.

Urdu textual contents were explored (Daud, Khan & Che, 2017) for classification using the majority voting algorithm. They categorized Urdu text into seven classes i.e., Health, Business, Entertainment, Science, Culture, Sports, and Wired. They used 21,769 news documents for classification and reported 94% precision and recall. Dataset evaluated using these algorithms, Linear SGD, Bernoulli Naïve Bayes, Linear SVM, Naïve Bayes, random forest classifier, and Multinomial Naïve Bayes.

A framework (Kuila, Chandra Bussa & Sarkar, 2018) proposed a tweet classification system to rescue people looking for help in a disaster like a flood (Singh et al., 2019). The developed system was based on the Markov Model achieve 81% and 87% accuracy for classification and location detection, respectively. The features used in their system are (Singh et al., 2019):

• Number of words in a tweet (w);

• Verb in a tweet by (verb);

• Number of verbs in a tweet by (v);

• Position of the query by (Pos);

• Word before query word (before);

• Word after query word (after).

To classify Urdu news headlines (Ali & Ijaz, 2009) by using maximum indexes of vectors. They used stemmed and non-stemmed textual data for experiments. The system was specifically designed for text classification instead of event classification. The proposed system achieved 78.0% for competitors and 86.6% accuracy for the proposed methodology. In comparison, we used sentences of Urdu language for classification and explored the textual features of sentences. We have explored all the textual and numeric features i.e., title, length, last-4-words, and the combinations of these (for more detail see Table 1) in detail in this paper that were not reported ever in state-of-art according to our knowledge.

Twitter (Agarwal & Rambow, 2010) to detect natural disasters i.e., bush fires, earthquakes and cyclones, and humanitarian crises (Sakaki, Okazaki & Matsuo, 2010). To be aware of emergencies situation in natural disasters a framework work designed based on SVM and Naïve Bayes classifiers using word unigram, bi-gram, length, number of #Hash tag, and reply. These features were selected on a sentence basis. SVM and Nave Bayes showed 87.5% and 86.2% accuracy respectively for tweet classification i.e., seeking help, offering for help, and none. A very popular social website (Twitter) textual data was used (Hussain, 2008) to extract and classify events for the Arabic language. Implementation and testing of Support Vector Machine (SVM) and Polynomial Network (PN) algorithms showed promising results for tweet classification 89.2% and 92.7%. Stemmer with PN and SVM magnified the classification 93.9% and 91.7% respectively. Social events (Usman et al., 2016) were extracted assuming that to predict either parties or one of them aware of the event. The research aimed to find the relation between related events. Support Vector Machine (SVM) with kernel method was used on adopted annotated data of Automated Content Extraction (ACE). Structural information derived from the dependency tree and parsing tree is utilized to derive new structures that played important role in event identification and classification. The Tweet classification of the tweets related to the US airlines (Rustam et al., 2019) is performed by the sentiment analysis companies that are not related to our work. We tried to classify events at sentence level that is challenging since the Urdu sentence contains very short features as compared to a tweet. It is pertinent to mention that the sentiment classification is different from the event classification. Multiclass event classification is reported (Ali, Missen & Husnain, 2021) comprehensively, deep learning classifiers are used to classify events into different classes.

Materials & methods

Event classification for Urdu text is performed using a supervised machine learning approach. A complete overview of the multi-class event classification methodology is given in Fig. 1. Textual data classification possesses a lot of challenges i.e., word similarity, poor grammatical structure, misuse of terms, and multilingual words. That is the reason, we decided to adopt a supervised classification approach to classify Urdu sentences into different categories.

Data collection

Urdu data were collected from popular social networks (Twitter), famous news channel blogs i.e., Geo News (https://urdu.geo.tv/), Urdu Point (https://www.urdupoint.com/daily/), and BBC Urdu (https://www.bbc.com/urdu). The data collection consists of the title, the main body, the published date, the location, and the URL of the post. In the phase of data collection, a PHP-based web scraper is used to crawl data from the above-mentioned social websites. A complete post is retrieved from the websites and stored in MariaDB (database). Our dataset consists of 0.1 million (102,960) label sentences of different types of events. All the different types of events used in our research work and their maximum number of instances are shown below in Fig. 2.

There are twelve different types of events that we try to classify in our research work. These events are a factual representation of the state and the situation of the people. In Fig. 2 imbalances number of instances of each event are given. It can be visualized that politics, sports, and Fraud & Corruption have a higher number of instances while Inflation, Sexual Assault, and Terrorist attacks have a lower number of instances. These imbalanced numbers of instances made our classification more interesting and challenging.

Multiclass events classification tasks are comprised of many classes. The different types of events that are used in our research work i.e., sports, Inflation, Murder & Death, Terrorist attacks, Politics, Law and Order, Earthquake, Showbiz, Fraud & Corruption, Weather, Sexual Assault, and Business. All the sentences of the dataset are labeled by the above-mentioned twelve (12) different types of events. Finally, a numeric (integer) value is assigned to each type of event label (See Table 2 for more details of the label and its relevant numeric value).

Table 2:

Types of events and their labels in the dataset.

Event	Label	Event	Label
Sports	1	Earthquake	7
Inflation	2	Showbiz	8
Murder and Death	3	Fraud and Corruption	9
Terrorist Attack	4	Rain/Weather	10
Politics	5	Sexual Assault	11
Law and Order	6	Business	12

DOI: 10.7717/peerj-cs.775/table-2

Experimental setup

Classifiers are the algorithms used to classify data instances into predefined categories. Many classifiers exist that process the textual data using a machine learning approach. In our research work, we selected the six most popular machine learning algorithms i.e., Random Forest (RF) (Livingston, 2005), K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Decision Tree (DT), Naïve Bayes Multinomial (NBM), and Linear Regression (LR).

Results

To evaluate our dataset, the Python package scikit-learn is used to perform event classification at the sentence level. We extracted the last four words of each sentence and calculated the length of each sentence. To obtain the best classification results we evaluated six machine learning classifiers among others i.e., Decision Tree (DT), Random Forest (RF), Logistic Regression (LR), Support Vector Machine (SVM), k-Nearest Neighbor, and Naïve Bayes Multinominal (NBM).

We proposed three features i.e., Length, Last-4-words, and Length and Last-4-words to classify sentences into different types of events (see Table 2). The results were obtained using ‘length’ as the feature is shown in Table 4. The classifiers i.e., DT, RF, NBM, and LR showed 32% accuracies that is very low. The comparatively second feature that is Last-4-words showed better results for these above-mentioned classifiers. Random Forest showed 52% accuracy that is a considerable result as an initiative for multiclass event classification in the Urdu language text. The detail of results regarding other classifiers can be seen in Table 5.

We also evaluated these classifiers using another feature that is the combination of both Length and Last-4-grams. It also improved the overall 1% accuracy of the proposed system. The Random Forest showed 53.00% accuracy. The further details of accuracies of other used machine learning models can be seen in Table 6.

The results obtained by using the above features are very low, we deiced to use the title of the post as a feature to improve the performance of the system. We integrated the “Title” of the post with each sentence of the same paragraph that dramatically improves the accuracy of the system. We combined the “Title” of the post with other features i.e., length, and Last-4-words. The detail of the highest accuracies that is obtained by the combination of these features i.e., Last-4-words, length, and title are given in Tables 7 and 8. Random forest and k-NN showed the highest accuracies. The detail of the confusion matrix related to the proposed system (TP, FP, TN, FN) is also given in Tables 9 and 10.

The standard performance measuring parameters i.e., precision, recall, and f1-measure of Random Forest and k-NN classifiers using “Title and Last-4words” as features are given in Tables 11 and 12 respectively. Similarly other combinations of features i.e., “Title and Length” are used to enhance the accuracy of the system. The Decision Tree and Random Forest showed the highest results as compared to other classifiers for this specific combination of features. A detailed summary of the results related to Decision Tree and Random Forest is given in Tables 13 and 14 respectively.

We finally presented the comparison of four classifiers that showed the highest results in Fig. 3.

The semantics of the script written in the Urdu language is quite different from that of English and Arabic Language which causes the low performance of SVM and k-NN as compared to Random Forest.

Discussion

Event extraction and classification are tightly coupled with processing resources i.e., part-of-speech tagger (PoS), text annotators, and contextual insights. Meanwhile, the usage of local languages being highly preferred over social media is creating problems to analyze by existing tools. Urdu is one of those languages that have a considerable number of users and a huge bulk of data on social networks. It contains worthy insights that are necessary to process for different purposes like to improve security, to understand the intentions, trends, and mindset of people. We performed event classification written in Urdu language text on different social media like platforms. The evaluation of results is presented after analyzing multiple features i.e., Length, Last-4-words, Title, and combination of all these features converged our findings to conclude that length and Last-4-words are basic features to classify multiclass events but showed 53% accuracy. To improve the accuracy of the proposed system, we integrated “Title” as the feature with other two features i.e., Length and Last-4-words. The combination of “Title” with “Length and Last-4-words” improved the performance of the proposed system and showed the highest results as reported in the abstract.

As described in the dataset section that the dataset is imbalanced and contains multiple classes. To validate the accuracy of results not only TP, FP, TN, FP reported but also the standard performance evaluations parameters i.e., precision, recall, and f1-measure are reported in Tables 11–14.

Furthermore, extracting and classification of events from resource-poor language is an interesting and challenging task. There are no standard (benchmark) datasets and word embedding models like Word2Vec or Glove (Exists for the English Language) for Urdu language text.

Conclusions

A massive amount of Urdu textual data exists on social networks and news websites. Multiclass event classification for Urdu text at the sentence level is a challenging task because of the few numbers of words and limited contextual information.

The selection of appropriate features and approaches is necessary to classify multiclass events written in Urdu language text.

The deep analysis of the structure of sentences written in the Urdu language leads us to select these appropriate features i.e. title, length, the last four words, and a combination of all these features.

Experimental results showed that non-of single feature is capable to classify multiclass events. In contrast to the different combinations of these features i.e. title and last-4-words, title and length and last-4-words and length showed considerable results.

Count_ Vectorizer and TF-IDF feature generating techniques are used to convert text into (numeric) real value for machine learning models. Random Forest classification model showed 52% and 53% accuracy for last-4-words and combination of length and last-4-words.

The title is the key feature that can dramatically improve the performance of event classification models that works on a sentence level. Combining title with last-4-words and length showed the highest accuracies i.e. 98.00% and 99.00% for Random Forest and k-NN classifiers, respectively.

Future work

• In a comprehensive review of Urdu literature, we found a few numbers of referential works related to Urdu text processing. One of the main issues associated with the Urdu language research is the unavailability of the appropriate corpus like the data set of Urdu sentences representing the event; the close-domain PoS tagger; the lexicons, and the annotator, etc.

• There is a need to develop the supporting tools i.e., the PoS tagger, the annotation tools, the dataset of the Urdu-based languages having information about some information associated with the events, and the lexicons can be created to extend the research areas in the Urdu language.

• In the future, many other types of events and other domains of information like medical events, social, local, and religious events can be classified using the extension of machine learning i.e., deep learning.

• In the future grammatical, contextual, and lexical information can be used to categorize events. Temporal information related to events can be further utilized to classify an event as real and retrospective.

• Classification of events can be performed at the document level and phrase level.

• Deep learning classifiers can be used for a higher number of event classes.

Supplemental Information