A Context-Enhanced De-identification System

2.1. n-gram context embeddings

Figure 2 shows how CEDI produces context embeddings using n-grams. First, for each token x_t, n preceding tokens (x_t−n,…,x_t−1) are extracted. Then the randomly initiated token embeddings for the n-gram of tokens (V_xt−n,…,V_xt−1) followed by the token embedding V_xt are loaded. The loaded token embeddings are read sequentially by the corresponding RNN units. The output from the last RNN unit, of which the corresponding token embedding was trained for token x_t, is the forward context embedding for the token. The same process is repeated with the backward n-gram (x_t+n,…,x_t+1) followed by x_t to derive the backward context embedding. The final context embedding to be fed into the next layer is produced by concatenating the forward and backward context embeddings. In the following layer, the context embeddings can be used alone or concatenated with other embeddings such as character embeddings, pre-trained token embeddings, and even other context embeddings, depending on the task.

CEDI has several advantages over the prevailing approach. First, context embeddings using n-grams do not require sentence boundaries; thus, it is not affected by the problem of erroneous sentence boundary detection.

Second, it captures the context by receiving the input through a window of 2n + 1 words rather than through a varying size window delimited by sentence boundaries. The fixed size context window provides consistency in input representation between training and testing.

Third, since the input sequence moves over the input layer, each token is not associated with a single input node; rather, it traverses through all nodes, making the system learn the relation between the tokens with respect to their relative distances.

Fourth, CEDI can detect dependencies up to n words from either side of the current token word; thus, the size of the context window is adjustable according to the need. An optimal window size can be learned through cross-validation during parameter tuning or can be adjusted a priori according to the nature of the text. For example, the requisite width of the context window can be set wider for psychiatric reports than the window width for nursing notes since sentences in psychiatric reports are much longer and dependencies between tokens often require a wider context for entity recognition.

Fifth, CEDI’s approach allows connections between the current token and the information mentioned in the previous or next sentence. For example, co-reference of the subject of the sentence (e.g., “It happened…”) cannot be resolved using the prevailing approach but it is quite possible with CEDI.

Sixth, pre-trained embeddings suffer from OoV terms (terms that do not exist in the pre-trained embeddings) as well as from homonymy (words such as “bat” with identical lexical form but with different meanings); however, CEDI alleviates both problems via contextualization. In pre-trained word embedding approaches, embeddings of two homonymous words are identical even though they have different meanings. Since CEDI does not take the word as a singleton, isolated from everything else, it produces embedding representations for each input word based on its surrounding words; thus, homonymous words are associated with distinct embeddings.

In case of OoV terms, the prevailing practice is to assign them non-informative random token-level embeddings and to use character embeddings in addition to token-level embeddings. While original NeuroNER mitigated problems from OoV on i2b2 2014 dataset using character embeddings, the results of an earlier study on i2b2 2016 dataset [20] indicated that this particular strategy was insufficient for overcoming the OoV problems on this new dataset. CEDI associates every OoV word with their corresponding context embedding, providing meaning to the OoV word using the information of the neighboring words. Thus, the resulting representation of the word is not arbitrary but a function of the surrounding words.

Another context embedding model, ELMo, has been suggested by Peters et al. [31], and in our previous work [20], we showed how ELMo can improve the de-identification performance of NeuroNER. ELMo is based on bidirectional language models. While ELMo is trained using a separate system (usually in a different and more general context), CEDI derives n-gram context embeddings internally; thus, there would be no context shifting with CEDI.

ELMo has several limitations compared to CEDI. To utilize a pre-trained ELMo model, the user needs to process the entire dataset and extract ELMo embeddings before starting NER training. This extraction process took over 6 hours for us for 2014 i2b2 shared task. Because all individual tokens have different context embeddings depending on the sentence, the storage and memory requirements for ELMo can also be overwhelming. In the 2014 i2b2 shared task, the entire extraction process required 30.3 GB of RAM and the extracted ELMo embeddings consumed 4.1 GB of storage—almost 500 times as large as the entire 2014 i2b2 shared task dataset, which was only 8.1 MB. Moreover, the extracted ELMo embeddings cannot be applicable to other datasets. In other words, the user needs to repeat this extraction process for each dataset separately. In contrast, CEDI derives n-gram context embeddings internally, saving the user from ELMo’s prerequisites, a lengthy extraction process, and extra storage for embeddings.

2.2. Deep affix features

Proposed by Yadav et al. [53], deep affix features posit that the first few characters (the approximate prefix) and the last few characters (the approximate suffix) are semantically informative. This notion is particularly relevant to the medical domain. For example, new or uncommon drug names can be OoV words, which are difficult for neural network-based NER systems to capture because they cannot be included in the pre-trained token embeddings. Their labels are therefore predicted from random embeddings. However, if the drug name ends with “-parin” and shares the same suffix embeddings with existing drug names such as dalteparin, enoxaparin, heparin and tinzaparin, the suffix would indicate the nature of the word, and using suffix embeddings, we may be able to predict that the word refers to a heparin-like anticoagulant drug. To apply affix features to CEDI, the first n-characters (prefixes) and the last n-characters (suffixes) of each token are extracted. We excluded low-frequency prefix and suffix candidates from the set, since they would unlikely be true affixes. We assigned a randomly initialized embedding to each unique prefix and suffix. This embedding is concatenated with the biLSTM output over characters of the token and the pre-trained token embeddings for the token, and fed into the next biLSTM layer. Yadav et al. showed that deep affix features may improve overall performance of NER generally as well as in the medical domain such as SemEval 2013 task [3] and 2010 i2b2 clinical NER task [46] by utilizing three-character (n = 3) affixes with various thresholds for minimum admissible frequency counts (e.g., 10, 20, or 50).

2.3. Attention mechanism

Attention mechanism directs the focus of the entity recognition system to a part of the input that is pertinent for the prediction task. The attention model was originally proposed by Bahdanau et al. [2] for machine translation and spread to other NLP tasks. It has been recently applied to NER tasks as well [33,50,56], but it has never been used for de-identification.

Using the attention mechanism, the biLSTM output (generated from the features such as character embeddings and token embeddings) is multiplied by the attention weights, yielding a context embedding for each token. The weights correspond to the amount of attention that the particular input “deserves” for the current prediction. This context embedding is concatenated (additive models) [26,34] or multiplied (multiplicative models) [47,52] with the hidden state of the corresponding timestamp and produces probability vectors over labels d_t, yielding the final prediction for the token y_t.

Attention mechanisms may be utilized at multiple places in an NER system such as over character embeddings, over token embeddings, and over context embeddings. An attention mechanism can be applied over character embeddings to find out which characters in the token are most critical to represent the token at the character level. If applied over token embeddings, it increases the weights of the informative tokens of the sentence for more accurate label prediction. CEDI uses the attention mechanism over concatenation of character, token, and context embeddings.

2.4. Datasets and experiments

We test the de-identification performance of CEDI using datasets of 2006 i2b2 de-identification challenge (2006 shared task, henceforth) [44], 2014 i2b2/UTHealth shared task (2014 shared task, henceforth) [37], and 2016 CEGS N-GRID shared task (2016 shared task, henceforth) [36]. The 2006 shared task dataset contains 889 medical discharge summaries. The 2014 shared task dataset consists of 1,304 diabetic patient records. The 2016 shared task dataset consists of 1,000 psychiatric intake records. Table 1 summarizes the statistics of each dataset. As shown, 2014 dataset is 39% bigger than the 2006 dataset, which is the smallest. The 2016 dataset is twice as big as the 2014 dataset. The average sentence lengths of the 2006 and 2016 datasets are 54% and 42% longer than that of the 2014 dataset, respectively.

The 2006 dataset contains 8 categories of PII, whereas the 2014 and 2016 datasets contain the same 28 types of PII in 7 categories as defined by the organizers of the i2b2 shared tasks (i2b2-PII, henceforth). The i2b2 PII categories are more extensive than the 18 types of HIPAA PII for better privacy protection. Originally, 2006 shared task provides 75% of the entire dataset as a training set. 2014 and 2016 shared task provide 60% of the datasets as training sets. The remaining 25% and 40% were used as test sets.

We split the original training set into training and validation sets using 2:1 ratio, respectively. We trained CEDI on the training set while checking its performance and tuning its parameters on the validation set. After reaching to a state where more training would not improve the performance of the system further, we stopped training, applied the trained system to the test set, and evaluated the output of the system on the test set.

2.5. System structure

We tested the effectiveness of CEDI by integrating each component into the state-of-the-art de-identification system, NeuroNER, one by one. To exclude potential factors that may inadvertently affect the system performance, we used the original NeuroNER code, maintaining all its parameters.

Originally, NeuroNER uses character embeddings and pre-trained token embeddings as features. Token embeddings can be trained on general corpora such as Google News and Wikipedia as well as domain specific corpora such as PubMed. While domain specific resources contain useful knowledge closely related to the relevant task, general resources may contain more generic semantics of English [48]. Therefore, the training resource should be carefully selected. NeuroNER achieved its highest de-identification performance using GloVe comprising general token embeddings from English version of Wikipedia 2014 and English Gigaword Fifth Edition [30]. Therefore, CEDI incorporates both character embeddings and GloVe, as well as its newly added n-gram based context embeddings.

Then we investigated the use of prefix and suffix features on the training data using 5-fold cross-validation. We found that prefix features increased the overall de-identification performance, but suffixes did not. Therefore, we only included the prefix features in our final system.

We also examined the benefits of an attention mechanism on the de-identification task. We tested it over characters, n-grams, and concatenated embeddings (consisting of character embeddings, token embeddings, deep prefix features, and context embeddings). We did not use attention over the components of the concatenated embeddings because the concatenated embeddings represent tokens better than their individual components. We found that only the attention over the concatenated embeddings improved the de-identification performance; therefore, we only used the attention mechanism applied to the concatenated embeddings.

2.6. Training and hyperparameters

We tuned hyperparameters on the training set using 5-fold cross-validation. Our selection for the final hyperparameters was as following: n-gram size = 10; character embeddings dimension = 25; prefix embeddings dimension = 25; prefix frequency threshold = 20; token embeddings dimension = 100; hidden layer dimension for LSTM over characters = 25; hidden layer dimension for LSTM over prefixes = 25; hidden layer dimension for LSTM over n-grams = 256; hidden layer dimension for LSTM over concatenation of {character, token, prefix and context} embeddings = 100; attention layer dimension = 50; dropout = 0.5; optimizer = SGD; learning rate = 0.02; maximum number of epochs = 100; early stopping = 15.

2.7. Evaluation metrics

We use the official evaluation scripts from the shared tasks. The primary metrics of the tasks are micro-averaged precision (P), recall (R) and F1-score (F₁). The organizers of the shared tasks applied the metrics in two different ways, entity-based and token-based. The entity-based evaluation considers a true positive, only when the system successfully de-identifies an entire entity. In contrast, the token-based evaluation gives partial credit when a system de-identifies at least one token in an entity. It is debatable which evaluation method is more informative for de-identification purposes; however, we select entity-based evaluation, because it is stricter and commonly used for general NER tasks. Due to non-deterministic characteristic of neural network training, the performance from our suggested system can be different each time. We therefore repeat each experiment five times and report results that reflect the arithmetic mean of five runs per experiment. The statistical significance of the F1-score differences from the baseline model is calculated using the approximate randomization test [28] with 9,999 shuffles.

3.1. Performance

The results of CEDI experiments are summarized in Table 2. We use the original NeuroNER as our baseline and test the effect of adding each of the new components. As previously mentioned, NeuroNER uses character embeddings and token embeddings as features. The F1-scores of NeuroNER are 96.0, 91.7, and 88.1 for the 2006, 2014, and 2016 datasets, respectively. CEDI accomplishes F1-scores of 96.4 (+0.4), 92.2 (+0.7), and 89.5 (+1.4) on the 2006, 2014, and 2016 datasets, respectively. Adding deep prefix features on to CEDI (CEDI + D_P) further increases F1-scores to 92.8 (+0.4) and 89.7 (+0.2) for the 2014 and 2016 datasets. Adding attention mechanism to this (CEDI + D_P + Att) further increases the performance to 96.5 (+0.1) and 92.8 (+0.1) for the 2006 and 2016 datasets. All F1-score differences from the baseline are statistically significant at α level of 0.01.

3.2. Computational efficiency

One of the main challenges of using neural network de-identification systems is that the computational efficiency is measured not only in run time cost but also in energy cost and impact to the environment (e.g., Green Algorithms using carbon dioxide equivalent (CO₂e) measures [19]). Computational efficiencies of the original NeuroNER, CEDI, and NeuroNER + ELMo on i2b2 2014 training set are shown in Table 3. Training CEDI on the i2b2 2014 training set with 790 records on a GPU with 2,304 CUDA cores took 15 hours 13 minutes on average, compared to 8 hours 2 minutes for the baseline and 15 hours 35 minutes for NeuroNER + ELMo. Inference time per record was 0.2, 0.5, and 0.8 seconds for NeuroNER, CEDI, and NeuroNER + ELMo, respectively.

4.1. Error analysis

The addition of context embeddings improves the de-identification performance over the baseline. This is particularly true, where sentence boundary detection is incorrect, or the sentence boundaries in the original documents are unclear. For example, in these datasets, IDNUM and USERNAME tokens are often recorded in a different line from their PII. In those cases, systems typically recognize IDNUM and USERNAME as single token sentences, for which character embeddings and random token embeddings do not provide enough information for the correct prediction, but context embeddings do. Another example is tabulated forms, which are common in electronic health records, but on which sentence boundary detectors perform poorly. They generally separate headings from contents and put them into multiple sentences resulting in many single-word sentences and losing all context information. This is particularly problematic with numeric data, where sentences may contain only text such as ‘##/##’. In these cases, de-identification systems without context embeddings are prone to misclassifying the numeric tokens as DATEs. Because CEDI takes into account adjacent tokens in the previous and following sentences, it is able to find clues that lead to better predictions.

PII such as personal names and addresses are difficult to be recognized by symbolic AI systems that are not trained on the data. RNNs in general and CEDI in particular recognize such entities using the context around them. CEDI uses n-grams, where n specifies the width of the context that can be adjusted according to the problem domain and the task. Performance comparison of CEDI against the baseline on such PII types is shown in Table 4. CEDI improved the baseline performance on all PII types except STREETs. We suspect this was because of the small number of instances of this entity type, which was inadequate for training CEDI.

The results show that the deep prefix feature, D_P, increases precision but not recall. This benefit manifests itself more frequently on capitalized words than all-lowercase words. Since prefix features filter out false positive predictions, high-frequency prefixes are less common in proper nouns (named entities) and this feature helps distinguish capitalized words that are not named entities.

The performance due to the attention mechanism depends on the length of the input sentence. Attention mechanism performs particularly well on longer sentences, in which the other embedding systems do not perform as well. Its highest performance improvement was on HOSPITALs, which appear in relatively long and complete sentences.

4.2. Comparison with ELMo

We compared CEDI’s n-gram context embeddings to ELMo. The results are summarized in Table 5. CEDI generates an F1-score of 96.4 (+0.4), 92.4 (+0.7), and 89.6 (+1.5) on the 2006, 2014, and 2016 datasets, respectively. Whereas NeuroNER + ELMo generates an F1-score of 96.3 (+0.3), 92.8 (+1.1), and 89.0 (+0.9) on the 2006, 2014, and 2016 datasets. While ELMo achieves a slightly higher performance gain (+0.4) on the 2014 dataset based on F1-scores, CEDI performs better (+0.1 and +0.6 on the 2006 and 2016 dataset, respectively), using less effort, time, energy, and computational resources than ELMo.

We also experimented with integrating ELMo into CEDI (CEDI + ELMo). This achieved the highest F1-score at 96.7 (+0.7), 93.3 (+1.6), and 90.5 (+2.4) on the 2006, 2014, and 2016 datasets, respectively. This implies that n-gram context embeddings generate complementary knowledge to ELMo embeddings and that understanding both general context and task specific context is helpful to increase the de-identification performance of the NER system.

4.3. n-gram context embeddings on other tasks

The problems from erroneous sentence boundary detection and co-reference across sentence boundaries are not limited to de-identification, but pertinent for all entity recognition tasks in the medical domain. The n-gram context embeddings which alleviate the problems may improve the system performance on other NER tasks. Therefore, we examined the usefulness of n-gram context embeddings on additional medical entity recognition tasks. We selected two datasets from previous shared tasks in the medical domain, the ShaRe/CLEF 2013 eHealth [38] Task 1 dataset (ShaRe/CLEF dataset, henceforth) and the fifth BioCreative challenge evaluation workshop [49] Track 3 CDR dataset (BC5CDR dataset, henceforth). The ShaRe/CLEF dataset contains disorder mentions from 299 de-identified clinical notes and BC5CDR contains disease mentions from 1,500 PubMed titles and abstracts. We set the training parameters identical to the ones we used for de-identification but increased the size of the n-grams to 20 for the BC5CDR dataset because biomedical literature requires larger context than clinical notes. As shown in Table 6, CEDI significantly improves entity recognition performance (p < 0.01) on both datasets. This implies n-gram context embeddings can be widely adapted as a feature for NER in the medical domain.

4.4. Implication of performance gain

As shown in Section 3, the performance of the baseline neural network de-identification is close to perfect. Only a small room for improvement (4.0%, 8.3%, 11.9% in F1-measure for 2006, 2014, and 2016 datasets, respectively) remained. Although the absolute numbers are small, the remaining gaps include patient names and locations such as street addresses, and are important for both clinical utility of the records and for the privacy of the patients. CEDI closed the performance gap by 10% (10%, 8%, and 13% for 2006, 2014, and 2016 datasets, respectively)—a bigger gain than ELMo alone. Training CEDI took similar time to training ELMo, but required less memory and storage. In inference, CEDI is more efficient than ELMo with 37% less execution time and 59% less environmental impact.