CN108021916A - Deep learning diabetic retinopathy sorting technique based on notice mechanism - Google Patents

Abstract

The present invention discloses a kind of deep learning diabetic retinopathy sorting technique based on notice mechanism, including：A series of eye fundus images are chosen as primary data sample, are divided into training set and test set after sample being normalized pretreatment, cutting；Parameter initialization and fine setting are carried out to main neutral net, training set image is inputted to main neutral net and is trained, generates characteristic pattern；The parameter of fixed main neutral net, notice network is trained using training set image, is exported lesion candidate regions degree figure and is normalized to gain attention and tries hard to, and will pay attention to trying hard to gaining attention power mechanism with characteristic pattern product；Notice mechanism is tried to achieve into result and inputs main neutral net, continues to train using training set image, finally obtains diabetic retinopathy grade separation model.By introducing notice mechanism, using diabetic retinopathy area data set pair, it is trained the present invention, and the information characteristics of lesion region are strengthened while network primitive character is retained.

Description

An Attention Mechanism-Based Deep Learning Diabetic Retinopathy Classification Method

technical field

The invention relates to a deep learning diabetic retinopathy classification method based on an attention mechanism, and belongs to the field of medical image processing.

Background technique

At present, clinical doctors diagnose diabetic retinopathy by observing and analyzing early pathological features on retinal fundus images, such as microaneurysms, hard exudates, and hemorrhages. In practice, there are many types of diabetic retinopathy, various lesions, and varying degrees of severity, making it difficult for ophthalmologists to diagnose. Therefore, in large-scale screening of diabetic retinopathy, computer-aided diagnosis technology can greatly reduce the burden on doctors, and quickly and effectively assist doctors to achieve classified diagnosis.

In most of the current automatic diagnosis algorithms, the classification of fundus images of diabetic retinopathy is mainly based on traditional manual methods to design and extract features, and then construct a classifier. For example, using manual features including shape, color, brightness and prior knowledge for diabetic retinopathy detection, these methods can only achieve better results on small data sets, due to the cumbersome manual feature extraction process, in large data sets low efficiency and poor robustness. With the development of artificial intelligence algorithms, some researchers have proposed a method for the classification and diagnosis of diabetic retinopathy directly based on deep learning. For example, the convolutional neural network is directly connected to fundus images for the classification task of diabetic retinopathy. Designed for all types of diabetic retinopathy, it only regards the convolutional neural network as a black box, and does not take into account the lesion distribution information closely related to the diagnosis, and lacks an effective and intuitive explanation.

Contents of the invention

Purpose of the invention: The purpose of the present invention is to address the deficiencies of the prior art and provide a deep learning diabetic retinopathy classification method based on the attention mechanism. This method introduces the attention mechanism into the deep convolutional network and embeds the attention network into In the deep network, it is trained with the diabetic retinopathy region dataset marked by experts. The attention network can introduce expert knowledge and generate a lesion region of interest containing the location of the candidate lesion region. This method can preserve the original characteristics of the network while Enhance the information features of the lesion area.

Technical solution: The deep learning diabetic retinopathy classification method based on the attention mechanism of the present invention is characterized in that it comprises the following steps:

(1) A series of fundus images in EyePACS dataset, DiaretDB1 dataset and Messidor dataset were respectively selected as the original data samples, and the fundus images were normalized and preprocessed, and then cropped to ensure the same size, and the cropped fundus images were Divided into training set and test set;

(2) Build a convolutional neural network model, which includes a main neural network and an attention network; use ImageNet parameters to train the main neural network separately, and fine-tune and save the main neural network by the parameters obtained through training Main neural network model parameters; among the saved main neural network model parameters, select the best main neural network model parameters for diabetic retinopathy grade classification to initialize the main neural network parameter part in the convolutional neural network, and initialize the remaining parameters randomly;

(3) Input the training set images in the EyePACS data set to the main neural network for training to generate feature maps; fix the parameters of the main neural network, use the training set images in the DiaretDB1 data set to train the attention network, and the attention network outputs a lesion Candidate region grayscale image;

(4) Normalize the grayscale image of the lesion candidate region generated by the attention network to obtain the attention map, and perform element point multiplication between the attention map and the feature map output by the main neural network, and the product obtains the attention mechanism;

(5) Input the results obtained by the attention mechanism into the main neural network, continue training using the training set images in the EyePACS dataset, adjust the parameters of the main neural network according to the set learning rate during training, and finally obtain the grade classification of diabetic retinopathy Model.

The EyePACs dataset is used as a training set for DR 5 classification. The DiaretDB1 dataset is a dataset of DR lesion areas marked by experts, which is used to train the attention network part. The Messidor dataset is another dataset of DR classification, which is used to verify the network. robustness.

Further improve the above-mentioned technical scheme, the preprocessing operation of the step (1) is: extract the foreground area of the image in the original data sample, and adopt the following formula to carry out normalized preprocessing,

I _c (x,y)=αI(x,y)+βGaussion(x,y,ρ)*I(x,y)+γ

Among them, I is the input image, * indicates the convolution operation, Gaussion(x,y,ρ) indicates the Gaussian filter with standard deviation ρ, and the parameters α, β, γ and ρ are empirically set to α=4, β=- 4. Taking the center of the fundus image as the center, corrode to 5% of the edge of the fundus image;

The preprocessed image is cropped to 720×720, and the images divided into the training set are randomly rotated 0°/90°/180°/270° to achieve image data augmentation.

Further, the method of transfer learning is adopted in the step (2), and the parameters trained on ImageNet are fine-tuned to the main neural network, and the best main neural network model parameters of the diabetic retinopathy grade classification are selected for the convolutional neural network. The main neural network is initialized, and the remaining parameters are randomly initialized.

Further, after normalizing the output of the attention network in the step (4), add 1 operation, the obtained result and the feature map of a specific layer of the main neural network are subjected to a dot multiplication operation, and then input into the main neural network for subsequent training. The feature layer is a conv2d_2b_3x3 convolutional layer, and the feature map is Feature map M.

Further, the attention network is a symmetrical fully convolutional network, including 5 convolutional layers in the downsampling process and 5 deconvolutional layers in the upsampling process.

Further, the main neural network adopts inception-resnet-v2, which integrates two modules of residual learning structure and perceptual structure.

In described steps (3) to (5), Nesterov Momentum algorithm is adopted as the optimization algorithm of convolutional neural network, and the momentum optimization factor used is 0.9 to update all weights; The loss function of grade classification of retinal lesions trains neural networks at all levels. In the process of error value backpropagation, the gradient value is calculated using the Nesterov Momentum algorithm, and the parameters of the network are updated by using the calculated gradient value. L2 regularization is used for all parameters of the network The weight decay factor is 0.0005; according to the loss function of the attention network, the Nesterov Momentum algorithm is used to calculate the gradient value, and the parameters of the attention network are updated to complete the iterative process of the network.

The present invention uses Nesterov Momentum algorithm (a kind of stochastic gradient algorithm) as optimization algorithm, uses mean square error (Mean Squared Error, MSD) as loss function training neural network at all levels at the same time, uses L2Weight Decay regularization to the parameter in the network, And use the batch-normlization method in the network. The optimization algorithm is applied to the training process of the network to update the network weight, calculate the gradient in the error backpropagation process of the network, and update the network parameters.

Beneficial effects: 1. Introduce the attention mechanism in the deep convolutional network, embed the attention network into the deep network, and use the diabetic retinopathy area data set marked by experts to train it. The attention network can introduce expert knowledge, A lesion region of interest is generated containing the location of the candidate lesion region.

2. Use a deep network based on the residual perceptual structure, and use the feature enhancement method in the training of the network to introduce the lesion candidate region knowledge generated by the attention network into the diabetic retinopathy classification task. This method enhances the feature information corresponding to the region of interest on the basis of retaining the original feature information.

3. Using the mean square error-based loss function in the network model to improve the diagnostic results of diabetic retinopathy. This model is a multi-task learning model, which can use expert knowledge to improve the classification performance, and at the same time, it can roughly locate the lesion area in the fundus image, and has good robustness.

Description of drawings

Fig. 1: Realization flow frame diagram of the present invention

Figure 2: Attention module structure diagram

Figure 3: Comparison of fundus images before and after normalization

Figure 4: Comparison of the lesion candidate regions generated by the attention structure network and the actual expert-labeled regions.

Detailed ways

The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

Embodiment 1: The attention mechanism-based deep learning diabetic retinopathy grade classification method provided by the present invention detects and recognizes the grade of diabetic retinopathy, and the specific operation is carried out as follows:

1. Select the data set;

(1) EyePACS dataset

The diabetic retinopathy five-category dataset contains 88,702 color fundus images from 44,315 patients, and the resolution of the images is between 1 and 2. The data set is divided into two parts: training set 35126 (from 17563 patients), test set 53576 (from 26788 patients). The DR severity level of each fundus image is marked by the doctor according to the ETDRS table: '0' means no diabetic retinopathy, '1' means mild non-proliferative diabetic retinopathy, '2' means moderate diabetic retinopathy, '3 'Indicates severe diabetic retinopathy,'4'indicates proliferative diabetic retinopathy. The EyePACS dataset has the following characteristics: (1) As shown in Table 1, the category distribution of the dataset is seriously unbalanced, in which category 0 accounts for 79%, category 3 and category 4 account for 2% and 4% respectively. 2%; (2) Since the collection of this data set comes from different fundus cameras, the resolution and image quality of the images are quite different.

Table 1 EyePACS data distribution

(2) DiaretDB1 data set

The data set has a total of 89 color fundus images, and each fundus image is manually marked by 4 medical experts with potential lesion areas, including microaneurysms, hard exudates, hemorrhages, and cotton wool spots. There are 5 images in this data set without diabetic retinopathy, and in the other 84 images, each fundus image contains at least one lesion area. The annotated label image is saved as a grayscale image of 0-255, where the larger the grayscale value, the more serious the lesion in the area.

(3) Messidor dataset

The data set has a total of 1200 color fundus images, including 540 normal images and 660 diseased images, including three resolutions: 1440x960, 2240x1488 and 2304x1536, and the images are in TIF format. According to microangioma, hemorrhage, hard exudate and other lesions, each image is marked with DR lesion grade from 0 to 3, among which there are 546 images with label '0' accounting for 46% of the total data volume, and 154 images with label '1' There are 247 images with label '2' accounting for 20.58% of the total data volume, and 253 images with label '3' accounting for 21.67% of the total data volume.

2. Data preprocessing

Because in the fundus image, apart from the region of interest that is useful for detection and diagnosis, the image also contains a large number of background regions, so before making a diagnosis, we first extract the foreground region for the content of interest.

I _c (x,y)=αI(x,y)+βGaussion(x,y,ρ)*I(x,y)+γ

The fundus image is normalized according to the above formula to obtain a preprocessed normalized image. Among them, I is the input image, * indicates the convolution operation, Gaussion(x,y,ρ) indicates the Gaussian filter with standard deviation ρ, and the parameters α, β, γ and ρ are empirically set to α=4, β=- 4. In the visual field, the edge of the fundus map usually has a halo phenomenon, with the center of the fundus image as the center, corroding to 5% of the edge of the fundus map. The normalized fundus image is shown in Figure 3.

In this method, the size of the input image used is selected as 720×720, and the preprocessed data set is augmented by random rotation (0 ^° /90 ^° /180 ^° /270 ^° ) and random flipping on the training set .

3. Training Convolutional Neural Network Model

During the network training process, the data is input to the network in batches. One batch of data contains two queues: the DiaretDB1 data set and the EyePACS data set. In each iterative training of the network: the DiaretDB1 data set is sent to the convolutional neural network. The attention network generates the grayscale image of the lesion candidate area, and the error of the attention network is calculated through the MSE loss function; the EyePACS data is sent to the convolutional neural network, and the corresponding lesion candidate area information is generated through the attention network, and the output of the attention network is normalized After conversion, add 1, and the obtained result is multiplied with the feature map Feature Map M of the specific layer conv2d_2b_3x3 of the main neural network, and then input into the main neural network for subsequent training; using mean-square error (mean-square error, MSE ) as a loss function for calculating the grade classification of diabetic retinopathy (DR) to train neural networks at all levels. According to the attention network loss function, update the parameters of the attention network; according to the DR level classification loss function, update the corresponding part of the main neural network parameters, and complete the sequential iteration process of the network.

The training of the attention network fuses the lesion region candidate map generated by the DiaretDB1 dataset with clinical information. This method can introduce expert knowledge into the network to improve classification performance. Figure 4 is an example of the comparison between the lesion candidate regions generated by the attention structure network and the actual expert-labeled regions.

During the training process, the present invention uses the mini-batch-based stochastic gradient descent method (Stochastic Gradient Descent, SGD) for optimization, and uses SGD with a momentum optimization factor of 0.9 to update all weights. The network uses MSE as the loss function, and uses the L2 regularization term for each parameter in the network, and the weight attenuation factor is 0.0005.

4. Analysis of processing results

This method uses the following four performance metrics to quantify the processing results, namely: classification accuracy (accuracy, ACC), specificity (specificity, SPE), sensitivity (sensitivity, SEN) and AUC (the area under ROCcurve) value. For the classification results, we added the use of quadratically weighted kappa values as another performance measure, which measures the direct agreement between the predicted value and the actual label. The kappa value is calculated as follows:

Construct an N×N confusion matrix O for the image prediction label, O _i,j corresponds to the number of images subscripted as (i,j), and the weighting matrix is expressed as:

EN _N×N represents the confusion matrix constructed by the real label of the image. Assuming that there is no correlation between the predicted label and the real label, the twice weighted kappa value is as follows:

The processing result of this method on the EyePACS dataset used has a kappa value of 0.840 on the validation set and a kappa value of 0.835 on the test set. Our method is further validated on the Messidor dataset. This method has achieved excellent results in various indicators of accuracy rate and AUC value. Specifically, this method achieved an accuracy rate of 91.6%, a sensitivity of 90.3%, a specificity of 95.2% and an AUC value of 0.963 on the referable/non-referable task.

In this embodiment, more than 30,000 fundus images marked by professionals are used for training, and an accuracy rate of 82% is achieved on more than 50,000 test data sets. A large number of experiments show that the deep learning method based on the attention mechanism proposed by the present invention has higher classification performance. The classification model established by the above-mentioned deep learning diabetic retinopathy classification method based on the attention mechanism can automatically grade diabetic retinopathy, and has good robustness on data with unbalanced category distribution, which is very popular in the medical field. is of great significance.

As stated above, while the invention has been shown and described with reference to certain preferred embodiments, this should not be construed as limiting the invention itself. Various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (7)

1. the deep learning diabetic retinopathy sorting technique based on notice mechanism, it is characterised in that including following step Suddenly：

(1) a series of eye fundus images in EyePACS data sets, DiaretDB1 data sets, Messidor data sets are chosen respectively to make For primary data sample, pretreatment is normalized to eye fundus image, cutting is carried out after pretreatment and ensures that size is identical, will be cut Eye fundus image afterwards is divided into training set and test set；

(2) convolutional neural networks model is built, the convolutional neural networks model includes main neutral net and notice network；Adopt Individually main neutral net is trained with ImageNet parameters, the parameter obtained by training is finely adjusted main neutral net And preserve main neural network model parameter；In the main neural network model parameter of preservation, diabetic retinopathy etc. is chosen Level classifies best main neural network model parameter to initialize the main neural network parameter part in convolutional neural networks, remaining Stochastic parameter initializes；

(3) the training set image in EyePACS data sets is inputted to main neutral net and be trained, generate characteristic pattern；It is fixed The parameter of main neutral net, trains notice network, notice network is defeated using the training set image in DiaretDB1 data sets Go out lesion candidate region gray-scale map；

(4) the lesion candidate region gray-scale map that notice network generates is normalized to gain attention and tried hard to, and by notice Into row element dot product, product gains attention power mechanism the characteristic pattern of figure and the output of main neural network；

(5) notice mechanism is tried to achieve result to input in main neutral net, using the training set image in EyePACS data sets after Continuous training, adjusts the parameter of main neutral net according to the learning rate of setting when training, finally obtains diabetic retinopathy etc. Level disaggregated model.

2. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：The pretreatment operation of the step (1) is：The foreground area of image in primary data sample is extracted, using following formula To pretreatment is normalized,

I_c(x, y)=α I (x, y)+β Gaussion (x, y, ρ) * I (x, y)+γ

Wherein, I is input picture, and * represents convolution operation, and Gaussion (x, y, ρ) represents the Gaussian filter that standard deviation is ρ, Parameter alpha, beta, gamma and ρ are rule of thumb arranged to α=4, and β=- 4, using eye fundus image center as the center of circle, erode to eyeground figure edge 5% region；

By pretreated image cropping be 720 × 720, and by be divided into training set image carry out 0 ° of Random-Rotation/ 90 °/180 °/270 ° to realize view data augmentation.

3. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：Using the method for transfer learning in the step (2), the parameter that training obtains on ImageNet to main nerve net Network is finely adjusted, and selects the best main neural network model parameter of diabetic retinopathy grade separation to convolutional neural networks Middle main neutral net is initialized, and random initializtion remainder parameter.

4. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：It will be carried out in the step (4) after the output normalization of notice network plus 1 operates, acquired results and main nerve The characteristic pattern Feature Map M of network certain layer conv2d_2b_3x3 carry out dot product operation, are then input to main neutral net It is middle to carry out follow-up training.

5. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：The notice network is a symmetrical full convolutional network, including 5 convolutional layers of down-sampling process and is above adopted 5 uncoiling laminations of sample process.

6. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：The main neutral net uses inception-resnet-v2, integrates two kinds of residual error learning structure and perceptual structure Module.

7. the deep learning diabetic retinopathy sorting technique according to claim 1 based on notice mechanism, its It is characterized in that：Calculated in the step (3) to (5) using optimization of the Nesterov Momentum algorithms as convolutional neural networks Method, the momentum Optimization Factor used are 0.9 all weights of renewal；In the repetitive exercise each time of network, using mean square error Loss function as diabetic retinopathy grade separation trains neutral nets at different levels, in error amount back-propagation process In, Grad, and the parameter of calculated Grad renewal network are calculated using Nesterov Momentum algorithms, L2 regularization terms are used to all parameters of network, weight decay factor is 0.0005；According to the loss function of notice network, Grad is calculated using Nesterov Momentum algorithms, the parameter of renewal notice network completes the iteration mistake successively of network Journey.