WO2021196632A1 - Intelligent analysis system and method for panoramic digital pathological image - Google Patents

Abstract

An intelligent analysis method and system for a panoramic digital pathological image. The method comprises: inputting a pathological image slice to be analyzed into an attention mechanism-based convolutional neural network model, extracting features of different scales of a pathological image, and fusing the extracted features to obtain fusion features; and inputting the fusion features into a trained pathological image classifier to obtain the classification result about whether the pathological image slice contains cancerization or not. According to the method, intelligent panoramic digital pathological image analysis is realized, the features of different scales are considered and the weight is given, so that the cost of manually analyzing and classifying the data is effectively saved, and the problem of excessive dependence on the technical level of doctors is avoided, the manpower and material resource cost of the doctors is greatly saved, and more cancer patients can be diagnosed and treated in time.

Description

Intelligent analysis system and method for panoramic digital pathological images 【Technical Field】

The present invention relates to the technical field of medical image processing, and more specifically, to an intelligent analysis system and method for panoramic digital pathological images.

【Background technique】

At present, tumor pathological diagnosis and later statistical analysis are based on the work experience and knowledge accumulation of pathologists to complete the analysis. The evaluation results are easily affected by subjectivity. There are many classifications of cancer subtypes, and some subtypes have similar characteristics, and a large number of pathology Manual data analysis is not only time-consuming, and fatigue can easily affect the analysis conclusions. According to the latest international clinical research results, manual statistical analysis of H&E tumor cell nuclei is prone to errors. The statistical error assessment of the percentage of nuclei is as high as 45%, and the analysis results vary greatly depending on the pathologist. For the same tumor, the operation The dynamic variation range of 10%-95% of inter-subject disparity can easily lead to false negative diagnosis results. The inaccurate analysis result will directly affect the patient's treatment plan, which brings great danger to the patient's life. Pathological examination is the current gold standard for clinical cancer diagnosis. The pathologist's cancer diagnosis mainly relies on visual inspection of tissue sample images captured by a microscope. However, for pathologists, they need to combine their long-term accumulated clinical analysis experience to judge whether there is cancer in the pathological section of rhinitis cancer. This method is not only time-consuming, but also requires extremely high professional knowledge of the doctor.

In recent years, with the rapid development of artificial intelligence technology, Computer Aided Diagnosis (CAD) has achieved great success in the medical field. The main methods of CAD in the diagnosis of pathological images include traditional machine learning and the more popular deep learning in recent years. Traditional machine learning requires manually extracting image features and then classifying them through a classifier. The analysis effect of this method mainly depends on the effect of manual feature extraction in the early stage. Compared with traditional methods, deep learning does not require manual feature extraction, can automatically dig deep features of pathological images, and directly perform end-to-end optimization. Although CAD technology has achieved a lot of success in the field of pathological images, in the actual algorithm construction, the analysis is still based on the characteristics of a single scale, and the characteristics of different scales are ignored.

In short, the main problems of pathological image analysis are: pathologists’ diagnosis takes a long time and requires high professional abilities; traditional machine learning methods to analyze pathological images mainly rely on the effect of extracting features , Which requires higher professional knowledge of researchers. The existing feature extraction based on deep convolutional networks is mainly for a single feature, and less consideration is given to features under different magnifications. However, the actual clinical diagnosis is all analyzed under different scales of the image.

[Summary of the invention]

The purpose of the present invention is to overcome the above shortcomings of the prior art and provide a panoramic digital pathological image intelligent analysis system and method, which adopts a deep multi-scale feature convolution network based on the attention mechanism to classify the digital panoramic pathological image to solve the pathology Due to the large scale and complex shape of the panoramic image, the feature information is not fully represented, and the automatic intelligent analysis of the digital pathological image is finally realized.

According to the first aspect of the present invention, a method for intelligent analysis of panoramic digital pathological images is provided. The method includes the following steps:

Input the pathological image slice to be analyzed into the convolutional neural network model based on the attention mechanism, carry out the features of different scales of the pathological image, and fuse the extracted features to obtain the fused features;

The fusion feature is input to a trained pathological image classifier to obtain a classification result of whether the pathological image slice contains cancer.

In one embodiment, the fusion feature is obtained according to the following steps:

The pathological image slices to be analyzed are sequentially input into each feature layer of the attention mechanism-based convolutional neural network for feature extraction of the corresponding scale, and the features of each layer are processed to have the same resolution and channel;

Cascade and merge the extracted features to obtain fusion features;

The fusion features are passed through a fully connected layer and then input to the trained pathological image classifier to obtain various probabilities.

In one embodiment, the pathological image slice to be analyzed is obtained according to the following steps:

Cut the pathological image into two-dimensional slices with preset dimensions;

The sliced pathological image block is input to the deep convolution network, and after a convolution operation, three-dimensional features are obtained.

In one embodiment, in the process of extracting features of different scales, the feature calculation model is expressed as:

Among them, "*" is the two-dimensional convolution operation, N _out is the output feature vector, N _in(k) is the feature vector of the k-th channel of the _{previous layer, C in} is the total number of channels of the previous layer, w _k Is the weight of the k-th channel, and bias is the bias value.

In one embodiment, the loss function for training the pathological image classifier is expressed as:

Among them, P _{(i, j)} represents the classification result of the pixels of the input image, P _{class (i, j)} represents the classification label of the input image pixel, weight (class) represents the specified weight of the category, C _in represents the total number of channels, and k represents For the k-th channel, weight(w _k ,k) represents the weight of the attention mechanism obtained for each channel, (i,j) represents the position of each pixel, and W and H are the length and width of the image, respectively.

In one embodiment, the classification result includes cancer, benign tissue, and inflammation.

In one embodiment, the pathological image slice to be analyzed is a noise picture with black, white, and red pixels randomly added according to the pixel size of the picture.

In one embodiment, the classification result is various types of probability information obtained by softmax.

According to the second aspect of the present invention, an intelligent analysis system for panoramic digital pathological images is provided. The system includes:

Feature extraction unit: used to input the pathological image slice to be analyzed into the convolutional neural network model based on the attention mechanism, carry out the features of different scales of the pathological image, and fuse the extracted features to obtain the fused feature;

Classification unit: used to input the fusion feature into the trained pathological image classifier to obtain the classification result of whether the pathological image slice contains cancer.

Compared with the prior art, the advantage of the present invention is that the proposed pathological image analysis of the deep multi-scale feature convolutional network based on the attention mechanism is a fully automatic analysis method, which does not require manual feature extraction of pathological images and avoids The analysis results are over-reliant on the knowledge of pathological image features, and different scale features are considered, and weights are automatically assigned accordingly, which can effectively save analysis time and improve analysis efficiency.

Through the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, other features and advantages of the present invention will become clear.

【Explanation of the drawings】

The drawings incorporated in the specification and constituting a part of the specification illustrate the embodiments of the present invention, and together with the description are used to explain the principle of the present invention.

Fig. 1 is a flowchart of a method for intelligent analysis of panoramic digital pathological images according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of a deep neural network model according to an embodiment of the present invention;

Fig. 3 is a flowchart of training a deep neural network according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of experimental results according to an embodiment of the present invention.

【Detailed ways】

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless specifically stated otherwise, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

The following description of at least one exemplary embodiment is actually only illustrative, and in no way serves as any limitation to the present invention and its application or use.

The technologies, methods, and equipment known to those of ordinary skill in the relevant fields may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be regarded as part of the specification.

In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, rather than as a limitation. Therefore, other examples of the exemplary embodiment may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further discussed in the subsequent drawings.

The intelligent analysis of panoramic digital case images provided by the present invention is a multi-scale pathological image recognition method based on the attention mechanism. In short, the method uses public pathology image data sets for training. The designed training network is a deep multi-scale feature convolutional network based on the attention mechanism. By extracting features of different scales, combined with the attention mechanism, learning through the network framework Corresponding scale weights are fused to obtain a richer feature expression of pathological images, so as to achieve accurate classification of pathological images. Further, the analysis of clinical pathological image samples can be realized according to the obtained training model.

Specifically, referring to FIG. 1, the method for intelligent analysis of panoramic digital images provided by the embodiment of the present invention includes the following steps:

In step S110, a training set is constructed based on the published pathological images of cancer tissues and clinical cancer data.

Collect and mark public cancer histopathological images and clinical cancer data sets containing different subtypes (for example, containing cancer, benign tissues such as lymph or surrounding tissues, inflammation, etc.), and construct pathological images of cancer tissues containing different subtypes Training set.

Step S120, preprocessing the pathological image.

In order to improve the accuracy of subsequent training, pathological images can be pre-processed, for example, noise images are added according to the original pixel size of the image to avoid the impact of slice quality (such as uneven staining) or irrelevant tissues such as blood vessels on the diagnosis. Noise pictures are mainly random single colors (black (0,0,0), white (255,255,255), red (255,0,0)).

In step S130, a multi-scale pathological image classification model based on the attention mechanism is constructed, and the constructed training set is used for training to obtain a pathological image classifier.

In this step, a multi-scale feature convolution pathological image classification model based on the attention mechanism is established, and the pathological image obtained in step S120 is used to perform deep learning training on different pathological image classification models.

In one embodiment, as shown in FIG. 2 and FIG. 3, the steps of the model training process include:

Step S131: Establish a multi-scale feature fusion pathological image classification model based on the attention mechanism

For example, referring to Figure 2, the established Attention-based Deep Multiple-scale Convolutional Neural Network (ADMCNN) includes multiple feature layers, denoted as ADMCNN56×56×64, ADMCNN28×28×128, ADMCNN14×14×256 and ADMCNN7×7×512.

Step S132, cut the obtained pathological image into pacth with a size of 224×224.

Step S133: Input the cut pathological image block to the deep convolution network, and after the first convolution operation, a feature with a size of 112×112×64 is obtained.

Step S134: Input the features obtained in step S133 into the established multi-scale pathological image classification model based on the attention mechanism, and extract the features through multiple convolutional layers.

In the process of extracting features, the parameters of the convolutional layer are used as the parameters of the attention mechanism. The extracted features are input to the next convolutional layer, so that the features of each layer have the same resolution and channel for subsequent processing. The corresponding feature calculation model is expressed as:

Among them, "*" is the two-dimensional convolution operation, N _out is the output feature vector, N _in(k) is the feature vector of the k-th channel of the _{previous layer, C in} is the total number of channels of the previous layer, w _k Is the weight of the k-th channel, bias is the bias value, input is the input, and weight is the weight.

In the embodiment of the present invention, in response to the need for clinical practice to consider multiple features of an image and each feature has a different weight, an attention mechanism is introduced to obtain a weight map corresponding to the scale, which is used to represent each scale. And the importance of pixels.

In step S135, these features are then cascaded to merge them into a feature with a size of 28×28×128.

In step S136, the merged features are passed through a fully connected layer.

In step S137, the features after the fully connected layer are input to the pathological image classifier, and various probabilities are obtained through softmax.

According to the above steps, the loss function model combined with the attention mechanism is designed as follows:

Among them, P _{(i, j)} represents the classification result of the pixels of the input image P _{class (i, j)} represents the classification label of the input image pixel, weight (class) represents the specified weight of the category, C _in represents the total number of channels, and k represents the k channels, weight(w _k ,k) represents the weight of the attention mechanism of each channel calculated after step S134, (i,j) represents the position of each pixel, W and H are the length and width of the image, respectively .

In this step, softmax is used to normalize, and the relative importance of each position pixel in each scale is obtained.

Repeat the above process to train the model until it converges to obtain a trained pathological image classifier.

Step S140, using the trained pathological image classifier to analyze the pathological image to be detected to obtain a classification result.

The trained pathological image classifier can analyze the target pathological image to be analyzed and obtain the classification result. Similar to the training process, the full clinical pathology slice to be analyzed is preprocessed, and the preprocessed pathological image is input into the deep convolutional network for feature extraction. Further, input the extracted features into the pathology classification network classifier to obtain the analysis result.

For further understanding, still in conjunction with Figure 3, taking the rhinitis cancer pathological image as an example, the model training, analysis and prediction process includes:

Step S310, collecting public rhinitis cancer pathological images containing inflammation, lymph and cancer;

Step S320, preprocessing the pathological image;

Step S330, establishing a multi-scale feature convolution pathology image classification model, and using the pathology image obtained after preprocessing to perform deep learning training on the pathology image classification model to obtain a reference pathology image classifier;

Step S340, acquiring digital pathological slice data of clinical cancer;

Step S350, the pathological image preprocessing similar to step S320;

Step S360, input the preprocessed pathological image into the deep convolutional network for feature extraction;

Step S370: Input the extracted features into the pathological image classifier, and obtain the analysis result.

It should be noted that the above-mentioned embodiments are only illustrative, and those skilled in the art can make appropriate modifications or changes without departing from the spirit of the present invention, for example, using other multi-classification functions in the prior art to replace the softmax function; Set up more convolutional layers in the multi-scale pathological image classification model based on the attention mechanism to extract more features of different scales; use other or modified loss functions to measure the accuracy of model training; cut the original training samples The transformed samples obtained after compression or stretching operations are used as the training set.

Correspondingly, the present invention also provides a panoramic digital pathological image intelligent analysis system, which is used to implement one or more aspects of the above method. For example, the system includes: a feature extraction unit, which is used to input the pathological image slice to be analyzed into the convolutional neural network model based on the attention mechanism, perform features of different scales of the pathological image, and fuse the extracted features to obtain Fusion feature; a classification unit, which is used to input the fusion feature to a trained pathological image classifier to obtain a classification result of whether the pathological image slice contains cancer.

In summary, compared to the prior art, the advantages of the embodiments of the present invention include: training on public data sets, adjusting clinical data for further learning, and finally testing on clinical data, which solves the limitation of the limited clinical labeling data set. Sex; uses the attention mechanism-based multi-scale feature convolutional network for feature fusion to obtain richer feature representations, which makes the analysis better; treats lymph as a separate category and classifies it, reducing the misclassification of lymph Is the misdiagnosis rate of cancer.

In order to further verify the effect of the present invention, the verification is performed on the nasopharyngeal carcinoma cancerous image, and a data set established by collecting pathological images from a 20-fold digital pathological slice is used as a test data set. Taking the ROC curve (receiver operating characteristic curve) of the classification problem of "whether the image contains cancerous area" as an indicator, the average time for analyzing a digital pathology full slice image (about 50,000 × 50,000 pixels at 20 times) is taken as The calculation complexity index is shown in Figure 4, where the ordinate is the true positive rate (True Positive Rate) and the abscissa is the false positive rate (False Positive Rate), respectively showing cancer (cancerous tissue) and lymph (lymph tissue). ), the AUC of inflammation (benign tissue) are 0.862268, 0.868492, 0.855835, respectively. It can be seen that the present invention can obtain accurate classification results for digital pathological images.

In summary, the present invention realizes fully automatic cancer analysis by constructing a pathological image classifier based on a deep multi-scale feature convolutional network based on the attention mechanism. The invention combines the actual clinical analysis steps from a technical point of view, considers characteristics of different scales and assigns weights, effectively saving the cost of manual data analysis and classification, and avoids the problem of excessive dependence on the doctor's technical level, and saves a lot of money The cost of manpower and material resources of doctors has been reduced, so that more cancer patients can be diagnosed and treated in a timely manner.

The present invention may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present invention.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .

The computer program instructions used to perform the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or in one or more programming languages. Source code or object code written in any combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect to the user's computer) connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions. The computer-readable program instructions are executed to implement various aspects of the present invention.

Here, various aspects of the present invention are described with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine that makes these instructions when executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner, so that the computer-readable medium storing the instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions onto a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , So that the instructions executed on the computer, other programmable data processing apparatus, or other equipment realize the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more components for realizing the specified logical function. Executable instructions. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.

The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the illustrated embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements of the various embodiments in the market, or to enable other ordinary skilled in the art to understand the various embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (10)

1. An intelligent analysis method for panoramic digital pathological images, including the following steps:

   Input the pathological image slice to be analyzed into the convolutional neural network model based on the attention mechanism, carry out the features of different scales of the pathological image, and fuse the extracted features to obtain the fused features;

   The fusion feature is input to a trained pathological image classifier to obtain a classification result of whether the pathological image slice contains cancer.
2. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein the fusion feature is obtained according to the following steps:

   The pathological image slices to be analyzed are sequentially input into each feature layer of the attention mechanism-based convolutional neural network for feature extraction of the corresponding scale, and the features of each layer are processed so that the features of each layer have the same resolution and channel;

   Cascade and merge the extracted features to obtain fusion features;

   The fusion features are passed through a fully connected layer and then input to the trained pathological image classifier to obtain various probabilities.
3. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein the pathological image slices to be analyzed are obtained according to the following steps:

   Cut the pathological image into two-dimensional slices with preset dimensions;

   The sliced pathological image block is input to the deep convolution network, and after a convolution operation, a pathological image slice with three-dimensional characteristics is obtained.
4. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein, in the process of extracting features of different scales, the feature calculation model is expressed as:

   

   Among them, "*" is the two-dimensional convolution operation, N _out is the output feature vector, N _in(k) is the feature vector of the k-th channel of the _{previous layer, C in} is the total number of channels of the previous layer, w _k Is the weight of the k-th channel, and bias is the bias value.
5. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein the loss function of training the pathological image classifier is expressed as:

   

   Among them, P _{(i, j)} represents the classification result of the pixels of the input image, P _{class (i, j)} represents the classification label of the input image pixel, weight (class) represents the specified weight of the category, C _in represents the total number of channels, and k represents For the k-th channel, weight(w _k ,k) represents the weight of the attention mechanism obtained for each channel, (i,j) represents the position of each pixel, and W and H are the length and width of the image, respectively.
6. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein the classification result includes cancer, benign tissue, and inflammation.
7. The intelligent analysis method for panoramic digital pathological images according to claim 1, wherein the pathological image slice to be analyzed is a noise picture with black, white, and red pixels randomly added according to the pixel size of the picture.
8. The intelligent analysis method for panoramic digital pathological images according to claim 2, wherein the classification result is various types of probability information obtained by softmax.
9. A panoramic digital pathological image intelligent analysis system, including:

   Feature extraction unit: used to input the pathological image slice to be analyzed into the convolutional neural network model based on the attention mechanism, carry out the features of different scales of the pathological image, and fuse the extracted features to obtain the fused feature;

   Classification unit: used to input the fusion feature into the trained pathological image classifier to obtain the classification result of whether the pathological image slice contains cancer.
10. A computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the steps of the method according to any one of claims 1 to 8.

Images