Writer verification of partially damaged handwritten Arabic documents based on individual character shapes

Paper contributions

The key contributions of this paper can be summarized as follows:

• Proposes and evaluates an individual character based text-independent writer verification approach for partially-damaged handwritten Arabic documents.

• Provides a comprehensive Arabic language dataset, which consists of 53,199 handwritten isolated Arabic characters, 8,144 Arabic words (which encompass all characters), and 10,780 character shapes extracted from these words. The extracted character shapes dataset is constructed through the manual extraction of every character from the set of Arabic words handwritten by multiple users.

• Provides a CNN based model to verify writers based on individual handwritten Arabic character shapes.

• Proposes a mechanism to identify the Arabic character shapes that are more effective for writer verification.

• Provides a comparative analysis of performance of writer verification based on isolated and extracted Arabic character shapes.

The rest of the paper is structured as follows. We start with an overview of the existing work being done related to writer identification in ‘Related work’. Then, we describe our proposed writer verification approach in ‘Proposed Approach’. It also describes the process used to develop our dataset. In ‘Experimental Results and Discussion’, we describe the experimental results and provide a discussion of these results. Finally, in ‘Conclusions’ conclusions are drawn. The list of abbreviations and symbols used in this paper are listed in Tables 1 and 2 respectively.

Isolated alphabet characters dataset (IAD)

In the Arabic language, words use different forms of the same character depending on whether the character occurred at the beginning, middle, end, or in isolation (regular) as shown in Fig. 1. Further, Arabic alphabet characters can be classified into different groups depending on their similarity in writing style as shown in Fig. 1. Since the main objective of this research is to verify the writer of Arabic text recovered from partially damaged documents, which might contain any of these character shapes, we chose to collect all variants (begin, middle, end, and regular) of one character from each group. Thus, the dataset consists of 65 different variants of Arabic character shapes across 18 groups. The data was collected from eighty two (82) different Arabic writers in a classroom setting. Each user wrote each character shape ten times. This resulted in a dataset of 53, 199 isolated characters from 82 different users. Figure 4 shows samples collected from three different users. In real-world applications, users can use different writing instruments. We had advised the students to use a ballpoint pen but did not restrict them to any specific color.

Extracted alphabet characters dataset (EAD)

The extracted characters dataset consisted of characters cropped from user handwritten Arabic text. The users were asked to write Arabic text (consisting of ten Arabic words). The set of words were selected such that they covered the entire set of Arabic alphabet characters (but not all character shape variations). Figure 5 shows a sample of user written text. The characters were extracted from these words manually, and a sample of extracted characters from the first word is shown in Fig. 6. The complete dataset consists of 10, 780 extracted characters from different users.

Figure 7 shows the proposed methodology for writer verification using individual characters. The input characters dataset was pre-processed to remove any surrounding whitespace, converted to grayscale so that ink color does not become a distinguishing feature. The image size was condensed to 64 × 64 pixels for reducing the number of overall features. The pre-processed dataset was then split into training, validation, and test sets with a 60-20-20 ratio. It was also made sure that each character shape variant is present with the same ratio in each set. In order to implement user verification, we posed the problem as one vs. rest (OVR) classifier with one binary classifier per user. The reason for selecting the OVR approach was that, in our problem domain we focused on writer verification rather than writer identification. The OVR approach is a suitable approach in our problem domain (a typical forensic analysis case), where user verification is sought from a small group of suspects and generally does not need hundreds of users.

Each user would then be verified by using its own model. The dataset for each binary classifier represented a single classifier’s training, validation, and test sets (with a 60-20-20 ratio) with two classes:

• target class (representing the target user of this classifier)

• rest class (represented the rest of the users).

Thus, the target class had fewer instances than the rest class. In order to balance the dataset, the target class data was augmented with a 5 percent random shift (left, right, up and down) along with a 10-degree random rotation. The CNN classifier was optimized using hyperparameter tuning to improve the validation accuracy. The trained models were then tested using each user’s test set to determine test accuracy.

Writer verification of partially damaged arabic documents

The individual writer verification models trained on isolated and extracted characters can then be used as components to verify authorship of damaged handwritten documents. It is a text-independent approach that can be used for any user written text. The approach works by extracting individual character (a_i) from each user written partially recovered text (w) where w = a₁, a₂, …, a_m. Each a_i ∈ w can then be used to verify the target user (user_j) using their corresponding character based model f_userj such that: (1)${\displaystyle f_{use{r}_j}\left(a_i\right)=\left\{\begin{array}{cc}1;\hfill & a_i\text{is verified to be written by}use{r}_j\hfill \\ 0;\hfill & a_i\text{is not verified to be written by}use{r}_j\hfill \\ \hfill \end{array}\right.}$

The verification accuracy (β_userj(w)) of the recovered text (w) for user_j can thus be computed as: (2)${\displaystyle {\beta }_{use{r}_j}\left(w\right)=\frac{\left(\sum _i{f}_{use{r}_j}\left(a_i\right)\right)}{\left|w\right|}}$

We can define a threshold ϕ such that a document with text (w) is verified to be written by user_j, if (β_userj(w) >  = ϕ). We will use the notation α_userj(w) to refer to the writer verification of text w for user_j: (3)${\displaystyle {\alpha }_{use{r}_j}\left(w\right)=\left\{\begin{array}{cc}1;\hfill & {\beta }_{use{r}_j}>=\varphi \hfill \\ 0;\hfill & \text{otherwise}\hfill \\ \hfill \end{array}\right.}$

We will use the notation β(w) to refer to average writer verification accuracy (across n users) of user handwritten text w: (4)${\displaystyle \beta \left(w\right)=\frac{\left(\sum _j{\alpha }_{use{r}_j}\left(w\right)\right)}{\left|n\right|}}$

In this paper we use ϕ = 0.75 (i.e., any handwritten text is verified to be written by user_j if 75% of recovered characters from the document are verified to be written by user_j). A proper selection of ϕ value is application-dependent. Figure 8 shows how our proposed writer verification approach can be deployed to verify the authorship of any document.

Writer verification using isolated characters

Our initial analysis was conducted using the IAD dataset. The purpose of the analysis was to determine the efficacy of CNN-based approach to verify a user based on their handwritten isolated Arabic characters. We started with a CNN model with a single convolution and neural network layer. Figure 9A shows the model accuracy with this configuration. It can be seen that model is not able to learn well from the data and both training and validation accuracies are quite low (about 50%). We then incrementally added convolution layers and neural network layers with increased filter sizes until overfitting occurred; the middle plot in Fig. 9B shows the model with over-fitting. The dropout layers were then added to reduce overfitting resulting in a better configuration with model accuracies shown in Fig. 9C. The optimized CNN model used for training purposes is shown in Table 5. The model takes as input 64 × 64 images and applies a convolutional layer with 128 filters (filter size 3 × 3). This is followed by an ELU activation layer to provide non-linearity and max pooling layer to extract prominent features and also reduce the features space. This was followed by three similar convolutional and max pooling layers. A dropout layer (probability = 0.5) was added after each max pooling layer to reduce overfitting.

The output of convolutional layers was 256 features that were then processed by a neural network hidden layer of 128 neurons followed by the output layer. Figure 10 shows the training, validation and test results for twenty randomly selected users’ OVR models based on the IAD dataset. The color-coding scheme is used to highlight the minimum, maximum and variation in the results. The model accuracy is represented as Ω. Therefore, Ω_{iad_training} column shows the training accuracy and Ω_{iad_validation} shows the validation accuracy during model training. It can be seen that average validation accuracy is 94% and the difference between training and validation accuracies is small. This indicates that the model has learnt quite well from the dataset.

The trained models were tested on the IAD test set. We represent recall and precision of ith model as γ_i and ρ_i respectively. Eqs. (5) and (6) show their calculations, where τ is the total number of correct target class predictions, ξ is the total errors made to verify the target class, and η is the total errors made by the model to incorrectly identify the other users as the target user. So, in essence, γ shows the verification accuracy of the target user (i.e., ratio of correct target class verification out of the target user written character shapes). Henceforth, we will use the term γ as target user verification accuracy. The metric ρ shows the ratio of correct target class verification out of all the target class predictions made by the model. (5)${\displaystyle {\gamma }_i=\tau /\left(\tau +\xi \right)}$ (6)${\displaystyle {\rho }_i=\tau /\left(\tau +\eta \right)}$

The column “test_iad_all” in Fig. 10 shows the test accuracy (Ω_test), precision (ρ) and target user verification accuracy (γ_{iad_test}) for testing the model against all characters in the IAD test set. In this paper, we are mainly concerned with target user verification accuracy (γ). The average γ_{iad_test} is 91%, which indicates that the trained model works reasonably well on previously unseen isolated characters to verify the target user. Some users had a low γ_{iad_test} values (e.g., user06 has 82%) while a few others had a very high value of γ_{iad_test} (e.g., 100% for user02). The very high validation and test accuracy attained by some users can be attributed to their unique writing styles.

In order to understand the reason behind the lower recall for some of the users, we had to look into the performance of the model on each user written character. We collected the ratio of verification errors made per character by each target user model. We represent the ratio of verification error made by ith target user model against kth character as δ_ik such that γ_i = 1 − ∑_kδ_ik. Figure 11 shows the average error (λ_k) across all users for each character for the IAD dataset where ${\lambda }_k=\frac{{\sum }_i{\delta }_{ik}}{n}$ where n is the total number of users. It can be seen that most of the characters got less than 10% error, but some characters (e.g., alif_regular, lam_regular, etc.) had high errors. For example, alif_regular had a 40% average error. It can be attributed to the writing style of these characters, as alif_regular is written like a straight line and there would be quite less distinction in its writing style across users.

The average errors (λ_k) shown in Fig. 11 do not provide us with enough details on whether the errors were made by a single user as an outlier or spread across a large set of users. In order to understand the distributions of errors, we show the individual error values (δ_ik) of two best, average and worst performing characters using heat map in Fig. 12. It can be seen that the best performing characters (kaf_regular and feh_begin) perform well across all users. The worst performing characters (alif_regular and alif_hamza) perform worst across the majority of the users.

Based on the above analysis, we can deduce that some character shapes have more distinguishing features while others have lesser distinguishing features for writer identification. Hence, it is better to ignore the worst performing character shapes for writer identification. We evaluated the model by eliminating the 25% worst performing character shapes (highlighted with bold font in Fig. 11). The results of Ω_{iad_reduced_test} and γ_{iad_reduced_test} are shown in the “test_iad_reduced” column in Fig. 10. It can be seen that the performance has improved for each user model with the reduced set of character shapes. The average model performance improved to 93.75% from 91.25%. The model trained on the IAD dataset performed quite well on the test set of isolated characters. However, in practice, we need to verify the writer based on written text rather than just the isolated characters. Therefore, we evaluated model performance on characters extracted from user written text by testing it against the test set of the EAD dataset. The column “test_ead” in Fig. 10 shows the Ω_{ead_test} and γ_{ead_test} values for the EAD test set. The average γ_{ead_test} was a meager 74% and six out of twenty users had γ_{ead_test} values close to 50%. This means that model trained on the IAD dataset does not perform well on characters extracted from the text. As anticipated, the isolated characters are quite different from extracted character and therefore cannot be used as a reliable model to predict user written text.

Writer verification using extracted characters

As seen in the previous experiments, the models trained on isolated characters cannot be used reliably to identify user written text (i.e., characters extracted from the text). Therefore, we evaluated a CNN based OVR model that was trained using the EAD dataset. The results obtained for twenty randomly selected users’ models are shown in Fig. 13. The average training and validation accuracies (Ω_{ead_training} and Ω_{ead_validation}) of these models was 97.5% and 92% respectively. This shows that the models learned well on training data. Test accuracy (Ω_{ead_test}) was also quite close to validation accuracy (89.2%). However, target user verification accuracy (γ_{ead_test}) was close to 85% which is lower than the target user verification accuracy of isolated characters (γ_{iad_all} = 91.3%). Upon further investigation, it was found that this can be attributed to the presence of large variations within the extracted character shapes for the same user. In contrast, the isolated character shapes of the same user did not have such a large variation. When users are writing words in a flow, the shape of the same character changes across words. The shape of the character also varies depending upon how the writer joins it with the neighboring characters. To illustrate these variations in the characters written by the same user, samples of two different character shapes (ain_middle and yaa_middle) written by same user (user05) are shown in Fig. 14.

It can also be noticed that some user verification models did not perform well, for example, user06 had target user verification accuracy of only 52.9%. On closer inspection, it was found that the model performed badly with more than 80% error on few characters (jeem_middle, feh_middle, ain_middle, noon_end, alif_hamza, lam_alif). For example, the average error on character “jeem_middle” from other user models was 13.1%, but the user06 model had an error of 94.7%. Similarly, character “ain_middle” had 97.4% error for user06 model while the average error for other users is only 12%. This large error is due to the resemblance of these characters with other users’ handwritten characters.

We used the same methodology as described in ‘Writer verification using isolated characters’ to identify the performance of individual characters. The average errors (λ_k) are shown in Fig. 15 and the 33% worst performing characters (having an average error larger than 17%) are highlighted in bold font. We re-evaluated the model using the reduced set of characters (i.e., using characters which are not highlighted with bold font in Fig. 15). The results of Ω_{ead_reduced_test} and γ_{ead_reduced_test} are shown in the “test_ead_reduced” column in Fig. 13. It can be seen that the performance has improved for each user model with the reduced set of characters. The average model performance improved to 87.3% from 85.3%.

It can also be observed in Fig. 15 that the similar shaped characters in EAD dataset have large variation in their errors. For example, we have noticed the disparities in average error of following similar shaped characters:

• yaa_begin (18.1%) and beh_begin (9.7%)

• Haa_middle (17.4%), Khah_middle (4.7%)

• Feh_begin (11.1%), Qaf_begin (5.3%)

• Teh_middle (17.6%), Theh_regular (12.4%)

• Noon_end (27.2%), Noon_regular(13.4%)

Upon further investigations, we discovered that sometimes a character shape written by a user matches with a different character shape of another user (when character shapes are similar). For example, we noticed that user09 did not put dots in the right place for yaa_begin. This resulted in a large error because the model took it as beh_begin of user07. Due to this reason, yaa_begin has larger error (18.1%) than beh_begin (9.7%). Similar observations were made for other similar shaped character in EAD. Noon_end has the highest error because of its high similarity in writing style across several users and also similarity with Noon_regular written by other users. Based on the above observations, it seems desirable to use only a single character from the group of similar character shapes during extraction process to reduce the overall chances of verification errors.

These issues were less prominent in IAD because of character shapes written in isolation being more consistent, accurate (e.g., no dots related issues) and our choice of using a single character shape from within the group of similar shaped characters (as shown in Fig. 1). This resulted in having lesser variations in character shapes written by the same user and also lesser chances of errors caused by similar shaped characters. For instance, for IAD dataset the errors for Yaa-begin, Beh-begin, Noon-begin are 6.3%, 5.4% and 7.2% respectively, which has relatively smaller differences in errors.

Using character based models for writer verification of partially damaged documents

In this section, we describe the evaluation of the character based models (as mentioned in ‘Writer verification using extracted characters’) for writer verification of partially damaged documents. Each character extracted from the recovered text w written by user_j was checked using the EAD model (f_userj(a_i)). The writer verification accuracy of recovered document (β_userj(w)) for twenty randomly selected users is shown in Fig. 16. The overall writer verification accuracy of the damaged documents (β(w)) is also shown at the end of the table.

These experiments were conducted using the Arabic text shown in Fig. 5. Experiments were conducted with varying percentages of recovered characters from the text (80% to 10% document recovery). For each recovered document, experiments were conducted varying the percentage of good performing characters in the recovered characters (10% to 90%). The set of good performing characters was based on Fig. 15 with an average error of less than 17%. The rest of the characters (shown in bold font in Fig. 15) were considered bad performing characters. Each experiment was repeated ten times with a random selection of good and bad performing characters from within the text.

It can be seen that overall user verification accuracy is highly dependent on the quality of characters recovered and it varies between 85% (with good performing character shapes =10%) to 95% (with good performing character shapes =90%). We get similar results when the percentage of a recovered document is smaller (say 10% document recovered). During forensic analysis, writer verification based on documents with higher recovered characters and higher overall accuracy would be preferred over writer verification based on lower recovery and lower overall accuracy. In some problem domains, forensic experts might be willing to trade off document recovery and only seek high-performance characters to increase the writer verification accuracy but in some other problem domains, experts might want to figure out verification accuracy based on the completely recovered document, irrespective of character shape quality.

Limitations and future work

The work has few limitations and therefore can be extended in multiple ways. Although our EAD dataset was based on a carefully selected group of Arabic words that covered all Arabic alphabet characters, it did not include all character shapes (i.e., begin, middle, end and regular variants). A more comprehensive extracted alphabet characters dataset should be considered to include all character shapes. Additionally, writer identification (rather than just verification) of partially damaged handwritten documents would be a more challenging task. Also, the extraction of character shapes from handwritten text is a manual and tedious process which limits the scalability of our approach. This can be overcome with some other techniques such as automatically identifying intact parts of the document using object detection techniques and then using existing CNN based approaches on the recovered documents. It would also be interesting to evaluate the impact of “transfer learning” on the accuracy of the developed models.