Restoration of fragmentary Babylonian texts using recurrent neural networks

Choosing Digital Akkadian Corpora.

One challenge in designing an automated text-completion tool is the limited data available in digital form. For similar unsupervised language modeling tasks in English, for example, one can collect practically endless amounts of texts online, and the main limitation is the computational challenge of storing and processing large quantities of data (7). For cuneiform texts this is not the case. Automatic optical character recognition cannot be used to reliably identify cuneiform signs, neither in their two-dimensional (2D) representations (hand copies) nor in three-dimensional (3D) scans of actual clay tablets (8–12). Various visual recognition algorithms are being applied to cuneiform, but the results are yet in their infancy (13–16). Therefore, one has to rely on a limited corpus of manually transliterated texts. Although Akkadian cuneiform texts span more than two millennia and the genres available for study are heterogeneous, for many periods, only a limited amount of digital text is available to train the learning algorithm.

Three temporally and geographically defined corpora in particular are well represented in the digital transliterations available today: the Old Babylonian (approximately 1900 to 1600 BCE; see ARCHIBAB website: http://www.archibab.fr/), Neo-Assyrian (approximately 1000 to 600 BCE; see State Archives of Assyria online website: http://oracc.org/saao/), and Neo- and Late Babylonian (approximately 650 BCE to 100 CE). Our choice of texts, however, was governed by a corpus-based approach so that we might exercise greater control over the diversity of text genres and phrasing. Therefore, we have decided to gather 1,400 Late Babylonian transliterated texts from Achaemenid period Babylonia (539 to 331 BCE) in Hypertext Markup Language (HTML) format from the Achemenet website (http://www.achemenet.com/).*

The Neo-Babylonian Corpus.

The corpus chosen is written in what is commonly termed the Late Babylonian dialect of Akkadian, attested from the rise of the Neo-Babylonian empire in 627 BCE until the end of the use of the cuneiform script, around the first century CE (17). We prefer to describe this corpus as Neo-Babylonian, in recognition of the continuity of the Akkadian dialect of Babylonia over the whole first millennium BCE.^† The largest number of texts known from this period are archival documents belonging to economic, juridical, and administrative genres (19–21). The main reason we expect our models to work well on these texts, despite the small amount of data, is that these tablets are official bureaucratic documents, e.g., legal proceedings, receipts, promissory notes, contracts, and so on. They are highly structured, usually short, and prefer parataxis over hypotaxis (see Fig. 2 for an example). These texts are tedious for humans to read and complete, but they display many patterns that are relatively easy for learning algorithms to model, which makes them ideal for our purpose. For a more detailed breakdown into text types and their structure and content, see Materials and Methods and the Dataset S6.

Algorithmic Background

In this section, we will give a very brief introduction to techniques for modeling language using RNNs (for a more detailed account, see ref. 6). We can view language as a series of discrete tokens $x_1,\dots ,x_T$, and our goal is to fit a probabilistic model for such sequences; i.e., we wish to find a parametric model that learns the distribution $p\left(x_1,\dots ,x_T\right)$ from samples. The first step is to use an autoregressive model, i.e., use the factorization $p\left(x_1,\dots ,x_T\right)={\prod }_{t=1}^{T}p\left(x_t|x_1,\dots ,x_{t-1}\right)$. What this means is that we can reduce the problem of modeling a full sentence to predicting the next token in a text on the basis of the tokens that precede it.

In RNNs, this autoregressive model $p\left(x_t|x_1,\dots ,x_{t-1}\right)$ is fitted using a hidden memory. Given the previous hidden memory $h_{t-2}$, the network first updates the memory based on the new input $x_{t-1}$ and then uses the updated memory to predict the next token and passes the updated memory to the next step. More formally:

where $x_{t-1}$ is a one-hot representation of the input token, $W$ indicates linear mappings, and $pro{b}_t$ is the vector of probabilities for each possible next token. This parametric model is trained by maximizing the training log-likelihood to produce the output model. While simple and effective, due to vanishing gradients simple RNNs have difficulties in modeling long time dependencies, i.e., situations in which the probability of the next token depends on information seen many steps before. To solve this issue, various modifications that introduce a gating mechanism, such as long short-term memory (LSTM), have been proposed (22).

As a baseline model for comparison we trained an n-gram model. The n-gram model is a model that assigns a probability to each token based on how frequently the sequence of the last $n-1$ tokens in the training set ended in that token. The main limitation of n-gram models is that for small n, the context used for prediction is very small, while for large $n$, most test sequences of that length are never seen in the training set. We used a 2-gram model, i.e., each word is predicted according to the frequency with which it appeared after the previous one.

Results

In order to generate our datasets, we collected transliterated texts from the Achemenet website, based on data prepared by F. Joannès and coworkers in the framework of the Achemenet Program (National Center for Scientific Research [CNRS], Nanterre, France) (http://www.achemenet.com/fr/tree/?/sources-textuelles/textes-par-langues-et-ecritures/babylonien). We designed a tokenization method for Akkadian transliterations, as detailed in Materials and Methods. We trained a LSTM recurrent network and a n-gram baseline model on this dataset (see Datasets S1–S3 for model and training details).

Results for both models are in Table 1. Loss refers to mean negative log-likelihood and perplexity is two to the power of the entropy (in both cases, lower is better).

As expected, the RNN greatly outperforms the n-gram baseline, and despite the limitations of the dataset, it does not suffer from severe over-fitting.

Discussion

Further study of the different restorations of the designed completion test, taking into account the full ranking of the answers, results in some interesting patterns. This qualitative analysis considers four categories for the answer ranking: 1) correct syntax (i.e., sentence structure), 2) correct semantic identification, 3) poor syntax, and 4) poor semantic identification (see Dataset S5 for the full data analysis).

The majority of the restorations, 44 cases, shows that the algorithm best identifies correct sentence structure (category 1: questions 1 to 23, 25 to 27, 29 to 35, 38 to 42, 46, and 48 to 52). Put more accurately, this means correct syntactic sequences of parts of speech based on the statistical frequency of smaller syntagmatic structures. A total of 34 restorations show correct semantic identification of noun class as well as related verbs in the answer ranking (category 2: questions 1, 3, 6 to 13, 16 to 18, 22, 24 to 27, 29, 31 to 33, 35 to 40, 42, 46 to 49, and 51). In fact, a large subset of these cases, 30 questions, shares also identification of correct sentence structure (i.e., both categories 1 and 2).

Good semantic identification probably derives from paradigmatic relationships between certain classes of words. For example, five cases possibly correctly identified usage of verbal forms based on their context (e.g., in direct speech; questions 3, 6, 18, 25, and 26). Take, for example, question 3: NAME ašú šá NAME ana NAME lú qíipi ébabbarra u NAME lú sanga LOCATION___ umma. The model ranked the four possible answers as follows: iqbi; liqbuú; bar; bán. The example not only shows a correct identification of sentence structure but also a recognition of the relationship between two different forms of the verb qabû, “to speak.” It does not necessarily reflect an understanding of verbal root form, however; the model’s success probably reflects the statistical frequency of iqbi in this context and the model’s recognition of its similarity to liqbuú. This statistical inference emerges more clearly in one of the mistakes made by the model in question 32, where it does not differentiate properly the grammatical person of the verb nadānu, “to give, pay” (taaddinu vs. inamdin).

The level of the model’s semantic knowledge becomes apparent with regard to noun class; 14 questions show possible correct identifications of countable nouns (questions 1, 9, 17, 29, and 33), names of professions (questions 11, 38, 39, and 46), temporal designations (questions 12, 36, and 41), gender (question 31), and even a contextual formulaic legal clause (the so-called elat-clause; question 51). Four cases show correct identification of prepositions, particle use, or pronouns (questions 7, 34, 42, and 51). The choice in question 7, between the related prepositions ina and ana, makes it clear that these choices are again based on frequency in specific contexts. Moreover, a purely statistical grasp of parts of speech seems to be a decisive factor in at least eight cases of restoration, which are best identified in the analysis as those fitting category 3: poor syntax (questions 24, 28, 36 to 37, 43 to 45, and 47). Such statistical inference achieves surprisingly good results—e.g., preferring kurkur over LOCATION after lugal (question 37)—with only one restoration ended up being erroneous (question 45). However, it is clear, by comparing this group to other restorations in the designed completion test, that statistical inference is not a consistently reliable method for choosing completions. For example, it can interfere with contextual identification of the correct restoration—by preferring ina igi over ina ${\mathrm{š}\text{u}}^{\text{II}}$ before NAME (question 35).

The model does not seem to identify alternate logographic and phonetic writings of the same words, e.g., Sum. da = Akk. itti, or Sum. im.dub = Akk. tuppi (questions 14 and 19). It obviously lacks enough examples of such interchangeability in the studied corpus, since it does identify when different logograms have similar usage (e.g., a and dumu, both meaning “son” or “descendant,” are the top answers in question 40). Further confusion can occur when the model detects a similarity between the answer and another word close by in the sentence, either a noun or a verb. Especially problematic are cases when there are very few similar sentences to train on, and so the algorithm makes an “educated” guess resulting in a mistake (for example, question 45).

Conclusion

Our model—as far as can be judged by this experiment—is, as expected, good in teasing out sentence structures. However, it was also surprisingly better than we expected in making semantic identifications on the basis of context-based statistical inference (rather than finding underlying grammatical rules and morphology). In order to greatly reduce the number of false identifications based on the statistical frequency of contextual semantic relationships, much more training material will be needed. Nevertheless, we have demonstrated that even without access to large amounts of data, we can successfully train LSTM models and use them to complete missing words. In our completion test, we show good results that, while not sufficient for fully automatic completion, prove that the model can be an invaluable tool in helping scholars with text restoration.

Our results with the Late Babylonian corpus are significant because most entry-level scholars or other interested historians and social scientists who focus on the large first-millennium BCE Babylonian archives cannot acquire the very specific knowledge and expertise to understand underlying political, social, or historical structures without reading through hundreds of texts. For this reason, we are in the process of incorporating our model into an online tool, called Atrahasis. It will be of immense help to scholars in the historical sciences, allowing them to overcome the high entry barrier to restoring fragmentary Akkadian texts. Initially, the model will achieve the most success with structured archival documents, but as the dataset grows, one can train the model on more genres, such as scientific or literary texts. Both access to the primary sources in their original state and the ability to restore broken passages are equally necessary for understanding Akkadian corpora on a macroscale.

Related Work

The task of text modeling and its applications in tasks such as text restoration, lies in the intersection of Natural Language Processing (NLP) and Computer Linguistics. The Computational Linguistics models are predominantly rule based, while current NLP models are predominantly statistical and use machine learning. Currently, machine-learning methods achieve state of the art performance on most NLP challenges. In most modern languages, basic text modeling tools can identify spelling errors such as missing, added, transposed, or wrong letters (23–26). These tasks can become more challenging when the language in question has a richer morphology, like Arabic, for example (27), or limited digital corpora (28), as in the case of ancient Near Eastern languages (29, 30).

One approach is to first parse the original text and then use this as an input for further tasks. To this end, many studies use rule-based models derived from grammar (31), finite-state machines (32), or lookup in a machine readable dictionary (33) or employ statistical models such as clustering algorithms (k-nearest neighbors or kMeans) (34). Most work on rule-based models has been done in Akkadian (see literature cited in ref. 35). The most recent study dedicated to Akkadian word segmentation used a combination of rule-based, dictionary-based, and statistical algorithms, with best results in dictionary-based models (60 to 80%) (35). Because its algorithms were fitted for East Asian languages like Chinese and Japanese, we concluded that a specific NLP model for Akkadian should be designed. Sumerian, with its simpler syntax, is in the center of the Machine Translation and Automated Analysis of Cuneiform Languages project, which employed dictionary- and rule-based models for annotation of Sumerian (36, 37). A similar study on Hittite designed rule-based models derived from grammar (38). Statistical/machine-learning models achieve better results overall, if they are tailored to the problem at hand. Our project, the Babylonian Engine, recently achieved state-of-the-art results for Akkadian prediction of sign and word transliteration and segmentation using NLP and machine-learning models on Unicode cuneiform: up to 97% using a BiLSTM Neural Network algorithm, see the web-tool Akkademia (https://babylonian.herokuapp.com/). A similar task of automatic phonological transcription of Akkadian (usually termed normalization) based on ORACC material has recently achieved promising results (39).

For the specific task of text restoration and prediction, statistical n-gram Language Models (LMs, e.g., bigrams, trigrams, etc.) are now widely used in NLP tools, including those designed for modern Indian languages (40). A bigram model was successfully applied to a hidden Markov model to restore missing or damaged sign sequences in the ancient undeciphered Indus script (41, 42). However, neural network LMs can perform better in developing meaningful patterns of representations of words and the contexts around them. When these “embeddings” are learned from unsupervised large corpora, they can be transferred to various tasks, retaining a boost in performance (43).

Specifically, RNN language models, like the one employed in this study, have shown success in encoding both semantic and orthographic data in languages of varying levels of morphological complexity (44). The best results in solving the restoration task, so far, have been achieved in studies of machine-reading comprehension, specifically of the cloze-style, in which both the level of character and word are identified (45). In the field of ancient languages, we know of only one other study that used an algorithm with neural network architecture to recover missing letters, in the context of epigraphic inscriptions in ancient Greek (46). Their model, named PYTHIA, uses a sequence-to-sequence NN with LSTM and was trained on the Packard Humanities Institute (PHI) database, the largest digital dataset of ancient Greek inscriptions. It gives best predictions with 30.1% character error rate, compared with the 57.3% error rate of human epigraphists (based on testing the performance of two doctoral students on the training material over 2 h).

Lastly, joining fragments of texts is one of the major challenges of restoring cuneiform manuscripts as close as possible to their original state. Matching fragments are usually only identified by a handful of experts, and the fragments are often so small as to retain only a few signs. An initial study on the Hittite corpus employed matching classifiers, achieving best results with Maximum Entropy Classifiers (47). The Electronic Babylonian Literature project aims to reconstruct the tens of thousands of fragments that make up the remnants of ancient Babylonian and Assyrian literature. A digital corpus of largely inaccessible tablet fragments from museum collections (15,510 fragments as of June 2020) allows users to query these fragments with sequence-alignment algorithms based on the word method Basic Local Alignment Search Tool algorithm. Initial results already show that one can identify new pieces of text as well as many possible text joins. Advances in cost-effective high quality 3D scanning allow exact measurements of inscribed objects that can lead to the joining of broken tablet fragments with a matching algorithm in 3D space, as done for example by the Virtual Cuneiform Tablet Reconstruction Project (48, 49).

Materials and Methods

Supplementary Material

Data Availability.

All study data are included in the article, SI Appendix, and Datasets S1–S6. Atrahasis can be accessed on the Babylonian Engine Website (https://babylonian.herokuapp.com/). Our source code is available at GitHub (https://github.com/DHALab/Atrahasis).