Automatic Staging of Cancer Tumors Using AIM Image Annotations and Ontologies

Ontological Representation of AIM Model

In order to perform inference and classify image annotations based on the AIM standard, we need a language equipped with formal semantic. Using this semantic, inferences about an ontology together with a set of individual instances of classes can be made. In this context, the Web Ontology Language (OWL), a language for building ontological representations of information models, was used. In our work, we transformed the AIM data model into an equivalent ontological representation, using OWL2. This transformation was performed by creating classes and properties in OWL that are user understandable and suitable for inference.

We developed an OWL model, based on the ontology provides by Bulu, Hakan and Rubin, Daniel L. [29]¹, which represented an older version of AIM. Therefore, in order to represent the AIM 4.0 model, which is the version used to store image annotations generated using tools (such as ePAD [30]), we modified Bulu’s ontology to represent AIM 4.0 model concepts. Our ontology is called AIM4-O.

AIM4-O Ontology

Creating Conditions to Implement Reasoning Based on TNM Rules Using OWL Instances

The second step is to transform existing AIM-XML documents to their equivalent in OWL (using the AIM4-O ontology). To achieve this, we developed scripts, in the Groovy language.² First, we automatically map AIM-XML entities to AIM Java classes, based on the AIM UML³ model (Fig. 4). We then create instances of the AIM4-O ontology from these AIM Java classes, using the OWL API, a Java API for creating, manipulating, and serializing OWL ontologies. Finally, these instances populate a semantic web knowledge base. This base is suitable for classification-based and rule-based inference.

In order to automatically stage cancer, our approach must have the support of an ontology to specify the semantics of image observations from a particular domain. In this case, this ontology should be able to represent the topology of a human body organ (the organ in which cancer starts growing and has its own TNM system). In this work, this organ was the liver. Furthermore, it is considered necessary to include an OWL representation of RadLexⓇ [31] vocabulary in order to facilitate handling AIM4-O individuals, because these individuals have Radlex terminology in their structure. Finally, a set of rules to do the actual staging, based on the TNM liver system, was added to the ontology. The final result was the General Ontology.

Formal Representation of TNM Cancer Staging

In the previous sections, several steps were necessary to create the General ontology for a TNM classifier. First, we created classes and properties in order to fill the semantic gap between the tumor features and the AIM4-0 classes definitions. Then, we provided formal definitions for the TNM stages, liver’s topological features, and RadLex terminology, in order to represent them in OWL. Finally, we defined formal mechanisms for reasoning (using only OWL and SWRL expressivity) such as OWL classes, intersections, equivalences, disjunctions between classes, and a set of rules in order to determine cancer stage from image annotations. This last step will be described below.

In order to discover the limits of the OWL concepts and SWRL rules, we attempted to formally define and implement the conditions that TNM staging demands. TNM cancer staging is divided in two main steps. The first step consists in giving a score starting from the description of the tumor (T), its spreading into lymphatic nodes (N), and possible metastasis (M) (see Table 1).

The second step consists in determining the stage according to the previous scores (see Fig. 6). To make the aforementioned tasks possible, we decided that the following conditions reflect a desirable staging process:

• Condition 1 : Staging should consider the existence of solitary or multiple tumors on the same site.

• Condition 2 : Staging should consider if tumors are either bigger or minor than a certain size in cm.

• Condition 3 : Staging should consider lesions in adjacent organs.

Asserting Conditions Using OWL

Condition 1:

Staging should consider the existence of solitary or multiple tumors on the same site.

The AIM4-O ontology does not give us the explicit mechanics such as classes, subclasses, or properties, that allows us to infer whether a patient has a single or multiple tumors. In the case of multiple tumors, we constructed the following rule MoreThanOneTumor (in SWRL notation):

This rule classifies an ImageStudy as a member of the MorethanOneTumor class if an image study “X” is referenced by more than one image annotation. In order to classify something as MorethanOneTumor, we created a new concept called isImageStudyOf. This concept is the inverse of the hasImageStudy object property. The hasImageStudy property relates an ImageAnnotation entity to an ImageStudy entity.

In the scenario of classifying patients with one solitary tumor, we did not find axioms or rules that satisfied this requirement, due to the fact that OWL works under the open-world assumption. Open world means that just because something is not said it does not mean that it is not true. For example, I can say that the patient annotation describes a cancer lesion, using the ImagingObservation entity of the AIM4-O ontology model, but unless I explicitly say that there are no other lesions, it is assumed that there may be other lesions that I just have not mentioned or described.

We have tried to solve this problem (the open-world assumption) by considering some alternatives such as modeling again our AIM4-O ontology (e.g., setting the hasImagingObservation object property as a primitive class). But, this did not seem intuitive to us. Instead, we decided to state the number of lesions explicitly by creating one new concept named singleLesion, as a data property of an ImageStudy entity. This concept denotes if an ImageStudy describes exactlyone solitary tumor. We assumed that an ImageStudy entity describes only one tumor ("singleLesion {true}") if and only if it is referenced by only one ImageAnnotation entity. However, it was not possible to formulate this using only OWL. Instead, this information was provided by a data structure that was generated as part of the process of parsing the AIM-XML image annotations to create AIM4-O individuals. Finally, to classify annotations that describe a single lesion, we constructed the rule SingleTumor (in SWRL notation):

Condition 2:

Staging should consider if tumors are either bigger or minor than a certain size in cm.

This condition was easily implemented by getting the value from the data property values on the CalculationResult entity. This entity is related to the ImageAnnotation entity through the hasCalculationEntity object property. In order to satisfy this condition, we assert the following rules taking 5 centimeters as the longest dimension of the target liver lesion (in SWRL notation):

LessThan5cmTumor:

MoreThan5cmTumor:

Condition 3:

Staging should consider lesions in adjacent organs.

To satisfy this condition, the most complicated criterion of classification, we had to consider the fact that a cancerous tumor can spread throughout the body. For that, we needed to create one new concept, based on the Lesion class from the Onlira ontology [32]. The Lesion class handles important characteristics of a lesion, such as composition, density, size, and shape. But, unfortunately, they are not enough for TNM classification and reasoning. For this reason, we added 3 properties to it and created the subclassOutsideLesion; these properties are:

• hasLocation (object property): This property indicates the lesions location based on RadLex taxonomy. This property relates Onlira Lesion class instances to RadLex AnatomicalEntity class instances (see Fig. 7).

• isRegionalLymphNodeAffected (data property): This property denotes whether a lesion is found in some lymph node. It was useful to enable classification criteria such as N0 and N1 (see Fig. 7).

• isAdjacentOrgan (data property): This property denotes whether a lesion with a hasLocation value ‘X'' is close to any adjacent organ. In accordance with the TNM liver classification criterion, which is the case of study in this work, we considered as adjacent organs to the liver [10]; the pancreas, duodenum, and colon (see Fig. 8). Furthermore, we grouped these concepts as organs in RadLex representation, creating two new classes, AdjacentOrganGroup and NoAdjacentOrganGroup:

AdjacentOrganGroup and NoAdjacentOrganGroup classes indicate whether a body organ is considered adjacent or not to the organ where the primary tumor was located. The primary organ defines the type of staging system to use; in our case, this organ was the liver. Finally, we constructed the following rule (in SWRL notation) to indicates whether an OutsideLesion is located in an adjacent organ:

Once the above requirements were adequately covered using OWL and SWRL rules, we constructed the axioms and rules in order to be able to automatically classify cancer lesions, based on the TNM system. We noticed that the way we modeled things mattered. For example, it was easier to define N1a and N0 criteria and reuse their definitions for M0, rather than to start with the definition of M0 and end up handling complex closures. With the use of the AIM4-O ontology, anatomical concepts can easily be related to each other as demonstrated previously.

Datasets

Our first dataset is a set of real clinical reports of Hepatocellular Carcinoma (HCC) patients from The NCI’s Genomic Data Commons (GDC). In this work, all experiments were supported by the GDC data. An important requirement to enable a feasible clinical evaluation was to have an image dataset to validate the results of the GDC clinical reports. To cover this requirement, we used the TCIA database [33]. It hosts a large archive of medical images about cancer, it is accessible for public download and it is related to the GDC records by a patient subject ID. The imaging modality selected was computed tomography (CT). The downloaded images were loaded into the ePAD annotation tool and annotated.

While TNM staging could be applied to other types of cancer, this work focuses on staging liver cancer. One reason was the availability of clinical data and images for this kind of cancer. For a given patient, the input to our TNM classifier consists of AIM files (image annotations) and the output consists of the Cancer Staging for this patient.

Quantitative Assessment of AIM4-O Ontology

According to Blomqvist et al. [34], “the ontological evaluation is the process of assessing an ontology with respect to certain criteria, using certain measures.” In this work, we undertook the evaluation of the AIM4-O ontology from the functional point of view. To achieve this, we carried out a task-focused assessment and inference requirements [32]. In order to evaluate the AIM4-O ontology, we studied and evaluated how it could help in searching clinical reports that describe image findings (reports about cancer). For this purpose, we compared two different approaches:

• Ontology-based (semantic) search: If the clinical reports are described as AIM4-O individuals, these reports can be searched using description logic query languages (DL query).

• Natural Language process-based (keyword) search: Clinical reports and image findings are usually written in natural language. There are many ways to implement keyword search. We decided to use a very popular full-text search engine that can be used from various programming languages: the Apache Lucene.⁶

In the literature, ontology-based search performs better than keyword-based search [32]

One of the reasons is that ontology search can search for information not explicitly mentioned in the text. For instance, it is possible to search for reports not having some features: Find all records of tumors not in the liver. Using keyword search, the best one can do is to find reports without the word liver. Many reports with the word liver may talk about tumors in other organs.

If an ontology-based search system, using the AIM4-O ontology, outperforms a keyword search system implemented using Apache Lucene it is a good positive quantitative assessment for the ontology.

In order to highlight the differences between the two approaches, we used four queries expressed both in DL (DL query) and keywords (see Table 2):

1. Q1—Find all reports related to an image observation (tumor observation).

2. Q2—Find all reports that describe multiple tumors.

3. Q3—Find all reports that contain a tumor observation that has a size greater than 8 cm.

4. Q4—Find all reports that contain a tumor observation with descriptors (e.g., invasion, mass, vascular).

The DL queries were processed using the ontology editor Protégé⁷ (using its default reasoner HermiT). For the keywords, a small Java program was created to read the report texts, read the keywords and use the class StandardAnalyser⁸, from the Apache Lucene library, to make the search.

We have considered the following points in order to evaluate both approaches:

• The evaluation was based on GDC reports: We randomly took 15 radiology reports of different patients written in natural language and converted them into AIM4-O instances.

• A report was retrieved if it satisfied the DL query or it collects all keywords in the search query.

• Finally, we compared the precision and recall against a gold standard. Precision is the proportion of truly retrieved reports to the total number of reports retrieved. Recall is the proportion of truly retrieved reports to the total number of reports that should have been retrieved [32].

The gold standard was determined manually by 3 radiology professors from two different institutions: one from Stanford University School of Medicine and two from the Faculty of Medicine of Marilia (Brazil). They manually evaluated each query to decide which of the 15 reports should be retrieved.

The four queries with corresponding precision and recall results are shown in Table 3.

By

analyzing the four queries, we can see that the semantic search has the greatest number of relevant documents retrieved:

Q1:

With the semantic approach, 15 reports were retrieved with an average precision of 0.95 and recall of 1.0. The keyword approach returned 12 reports with 1.0 precision and 0.96 average recall.

Q2:

With the semantic approach, 5 reports were retrieved with a 1.0 precision and recall. Much better than the 2 reports retrieved by the keyword with just 0.4 precision and recall.

Q3:

In this question, the keyword approach was very poor with no reports retrieved. There were no reports containing the queried words (i.e., “lesion size greater than 8 cm”). Keyword search is not well suited for queries with numerical relations, but such relations are very important when searching for tumors. The semantic approach returned 10 reports with 0.9 precision and recall.

Q4:

The semantic approach retrieved more reports (15 vs 7) with 0.67 precision and 1.0. When compared with the keyword approach (precision 1.0 and recall 0.7), it had a similar performance with 0.80 F1 versus 0.82 of the keyword search.

The semantic search approach performance was better, with recall values close to 1 and always better than the keyword search in both golden ratio values. Also, in all but one case, precision values were better for the semantic search. That shows that the AIM4-O ontology is able to semantically represent the information in the reports well enough to outperform keyword search. Its representation can also be used by reasoners to successfully compute information note explicitly stated in the report (such as the fact of a tumor be bigger than a given size, question Q3).

Automatic TNM Clinical Stage

In this section, we calculated the classification rate of the TNM classifier. At first, we created an ePAD template named “TNM template” in order to provide radiologists with a prespecified set of semantic terms for image annotations. These image annotations, which are compatible with the ePAD tool, were stored in the AIM-XML format. An example of an annotated image is presented in Fig. 9.

After, the generated image annotations (in AIM-XML format) were classified automatically using the TNM criteria. This process was duly evaluated and correctly accepted by two radiology professors from two different institutions (Stanford University School of Medicine and Faculty of Medicine of Marilia). They also analyzed the accuracy of the generated annotations in terms of semantics. The process we followed was:

• The data set used came from the following open databases:

  • The NCI’s Genomic Data Commons (GDC)⁹.
  • The Cancer Imaging Archive (TCIA)¹⁰ (collects only images, the number of series and studies): As we were working with TNM classification from the liver, we searched for “LIHC - Liver hepatocellular carcinoma” obtaining 52 patients with information available in both databases (images and reports). However, the information about tumor size was obtained by manual review of the medical reports. These reports are also available in The NCI’s Genomic Data Commons.

• After reading the medical reports, the radiologist was provided with an excel spreadsheet that provided information about medical findings, such as lesion size, vascular invasion, and others.

• Based on this excel file and the GDC data, we created AIM-XML annotations and integrated them into our knowledge base (as AIM4-O ontology individuals).

• The AIM files were used as inputs for our TNM classifier. The produced output was compared with the TNM values that physicians reported.

The AIM image annotations were generated based on 52 different clinical reports. Our automatic staging approach was evaluated by using precision and recall values. The cancer stages generated by our TNM classifier were compared to those described by the physicians who created the original clinical reports (our gold standard). We used the 7th edition of TNM [10]. One patient, with the subject ID “TCGA-DD-A1EJ,” was removed from this analysis. Our radiology professors considered that the TNM classification, reported by the physician in his respective clinical report, was incorrect (more information below).

For the calculation of precision and recall, the result is considered positive when the automatic staging coincides with the stage given by physicians who created the original clinical reports (see Table 4).

Precision was 85.7% and recall 81.0% (for 51 patients). This means that, for precision, at 85.7% of the time the system agreed with the staging given by physicians. For the recall, this means that, of all the times that a given stage was reported by a physician, in 81.0% of cases the system agreed with him/her. It is important to note that, even when the system diverged from physicians, the maximum difference between them was only one stage.

In Fig. 10 we show the results of the evaluation summarized in the color scale matrix. It represents our confusion matrix for a multi-stage classification. The darker the square in the diagonal of the matrix means that the respective class was better classified. The other squares in grays, outside of the diagonal, indicates that the class in the vertical axis was confused by the classifier with the corresponding class on the vertical axis.

For early stages of cancer, such as I, II, and IIIA, the percentage of misclassifications (e.g., false positives and false negatives) was very small. They are represented by the highlighted diagonal of the matrix (Fig. 10). For more advanced stages of cancer, such as IIIB or IVA, it was larger (Fig. 11). This may have happened simply because we had few patients at these stages or because these stages are described by relatively more complex concepts.

We also performed a sanity check of what was recorded in the AIM and the output produced by the classifier. The 3 radiologists that participated in this validation manually reviewed the whole process (including the stage assigned in the patient report), using the patient images and reports, for 3 randomly chosen patients. In all cases, the whole process for generating the AIM representation and staging classification was correct.

Our classifier also revealed the fact that there are clinical reports with inaccurate staging diagnosis. An example of this situation was the clinical case with subject ID “TCGA-DD-A1EJ.” This case was the only in which the difference between the classifier’s stage and the physician’s evaluation differed by more than one level, our radiologists decided to analyze how thus case was processed. They concluded that the case has been processed correctly and that the result of the classifier was also correct. The stage predicted was Stage I; however, the stage described by the medical report was Stage III.¹¹ They recommended us to not use this patient’s data, so this report was excluded from our analysis. Examples, like this, serve to warrant the importance of improving clinical decision support systems (through the use of image metadata in cancer treatment).