CN115984406B - SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling - Google Patents
SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling Download PDFInfo
- Publication number
- CN115984406B CN115984406B CN202310264537.8A CN202310264537A CN115984406B CN 115984406 B CN115984406 B CN 115984406B CN 202310264537 A CN202310264537 A CN 202310264537A CN 115984406 B CN115984406 B CN 115984406B
- Authority
- CN
- China
- Prior art keywords
- neural network
- image
- mask
- stripe
- sampling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005070 sampling Methods 0.000 title claims abstract description 74
- 238000007906 compression Methods 0.000 title claims abstract description 23
- 230000006835 compression Effects 0.000 title claims abstract description 23
- 238000013135 deep learning Methods 0.000 title claims abstract description 18
- 238000003384 imaging method Methods 0.000 title claims abstract description 11
- 230000003595 spectral effect Effects 0.000 title claims description 14
- 238000013528 artificial neural network Methods 0.000 claims abstract description 156
- 230000000295 complement effect Effects 0.000 claims abstract description 39
- 238000001514 detection method Methods 0.000 claims abstract description 12
- 230000000694 effects Effects 0.000 claims abstract description 12
- 238000013144 data compression Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000000034 method Methods 0.000 claims description 46
- 230000006870 function Effects 0.000 claims description 44
- 238000012549 training Methods 0.000 claims description 32
- 230000004913 activation Effects 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000012805 post-processing Methods 0.000 claims description 6
- 230000008439 repair process Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 abstract description 7
- 238000001228 spectrum Methods 0.000 abstract description 7
- 238000012014 optical coherence tomography Methods 0.000 description 57
- 238000010276 construction Methods 0.000 description 6
- 241000234282 Allium Species 0.000 description 5
- 235000002732 Allium cepa var. cepa Nutrition 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 210000004351 coronary vessel Anatomy 0.000 description 4
- 238000009795 derivation Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000126130 Ganymedes Species 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000004165 myocardium Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Pathology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of image processing, and discloses an SS-OCT compression imaging method for deep learning and spectrum domain airspace combined sub-sampling, which comprises the following steps: acquiring complete sample interference fringes and complete target detection interference fringes; step two: constructing and generating an image sampling system; step three: and inputting the complete target detection interference fringes into the image sampling system to obtain a target image. According to the invention, the mask neural network, the enhancement neural network and the stripe complement neural network are synchronously subjected to gradient descent update of parameters, so that the combined optimization of the two-dimensional mask and OCT image reconstruction is achieved, the problem of poor OCT image reconstruction effect caused by the adoption of an empirical fixed two-dimensional mask in the current OCT image sampling system is solved, and the OCT image reconstruction quality of the image sampling system is optimal under the same data compression ratio.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an SS-OCT compression imaging method for deep learning and spectrum domain airspace combined sub-sampling.
Background
Swept-frequency optical coherence tomography (SS-OCT) is a non-invasive volumetric imaging modality, widely used in the biomedical field. Thanks to advances in laser technology, SS-OCT imaging rates (typically quantified by the a-line rate of the scanning light source) are increasing all the way from a few hertz to a few megahertz. For a typical 200 kHz SS-OCT, the data bandwidth may exceed 800 MB/s if 2048 spectral sample points are measured in 12 bits. Researchers have proposed various strategies to mitigate data bandwidth. However, existing approaches mostly achieve reduced data bandwidth by employing a "sub-sampling and reconstruction" paradigm: the object interferograms are first spectrally or spatially sub-sampled and then OCT images are reconstructed by signal processing techniques.
In the method, a fixed sub-sampling mask is needed to reduce the data bandwidth, and the quality of the reconstructed OCT image is low because the fixed sub-sampling mask cannot be optimized together with the OCT image reconstruction.
For the problems in the related art, no effective solution has been proposed at present.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides the SS-OCT image compression method for the deep learning and spectrum domain airspace combined sub-sampling, which has the advantages of neural network combined synchronous optimization and the like, and solves the problems that a fixed sub-sampling mask cannot be optimized together with OCT image reconstruction, so that the reconstructed OCT image is low in quality.
In order to solve the technical problem that the quality of the reconstructed OCT image is low because the fixed sub-sampling mask cannot be optimized together with the OCT image reconstruction, the invention provides the following technical scheme:
an SS-OCT image compression method combining deep learning and spectral domain spatial domain joint sub-sampling comprises the following steps:
step one, constructing and training each neural network;
constructing a mask neural network:
the input of the mask neural network is random noise, two sets of feature images are output through a U-Net architecture of four layers of up-sampling and four layers of down-sampling, wherein each layer of up-sampling or down-sampling comprises two layers of convolution layers, and the two last sets of feature images generate a two-dimensional mask through GumbeSoftmax activation functions;
wherein GumbeSoftmax activation function is:
wherein D and
the output of the mask neural network and Gumbel Softmax activation function are respectively; random selection->
As a two-dimensional mask; g is random noise of independent and identical samples in Gumbel (0, 1) distribution, τ is a value controlling Gumbel distribution density, p and c are pixel index and channel index, respectively, # and (x) are the values of the Gumbel distribution density>
The output of the gummel Softmax activation function, represented as channel c, pixel p; />
The output of the mask neural network is represented as channel c and pixel p;
the output of the mask neural network is represented by the channel k and the pixel p; />
Output represented by random noise under channel k and pixel p, < >>
An output representing random noise with channel c and pixel p;
the method for constructing the stripe complement neural network comprises the following steps:
the input of the stripe complement neural network is undersampled stripe, and the complement sample stripe is obtained through the output of the U-Net architecture of four layers of downsampling and four layers of upsampling, wherein each layer of upsampling or downsampling comprises two layers of convolution layers;
the method for constructing the enhanced neural network comprises the following steps:
the input of the enhanced neural network is a preliminarily reconstructed image, and the enhanced OCT image is obtained through the output of a U-Net architecture of four layers of downsampling and four layers of upsampling, wherein each layer of upsampling or downsampling comprises two layers of convolution layers;
training the mask neural network, the stripe complement neural network and the enhancement neural network until convergence to obtain a trained mask neural network, a trained stripe complement neural network and a trained enhancement neural network;
the convergence time is the time when the similarity between the enhanced image and the true image reaches the standard;
the enhanced image is obtained by the following steps: inputting random noise into the mask neural network after the current iteration, and outputting and generating a two-dimensional mask; multiplying the two-dimensional mask with the corresponding point of the complete sample interference fringe to obtain an undersampled fringe; inputting the undersampled stripe into a stripe complement neural network after the current iteration to obtain a complement sample stripe; obtaining a preliminarily reconstructed image by carrying out IDFT on the completed sample stripes; inputting the preliminarily reconstructed image into an enhanced neural network after the current iteration, and outputting to obtain an enhanced image;
generating an image sampling system, wherein the image sampling system comprises a two-dimensional mask generated by outputting a mask neural network after training, a stripe complement neural network after training, an enhancement neural network after training and IDFT;
step three: and inputting the complete target detection interference fringes into the image sampling system to obtain a target image.
Preferably, the method for acquiring the complete sample interference fringes and the complete target detection interference fringes comprises the following steps: and acquiring the sample object and the object to be detected through the OCT system to obtain complete sample interference fringes and complete target detection interference fringes.
Preferably, the image truth value is obtained by: and obtaining a preliminary OCT image by the integral sample interference fringes and IDFT, and obtaining an image truth value by calculating the preliminary OCT image through a classical post-processing algorithm.
Preferably, the method of training the masking neural network, enhancing the neural network and stripe complement neural network comprises:
in the training framework, a batch size of 1 was used, an initial learning rate of 0.001, a network was trained for 30 cycles using an AdamW optimizer with momentum (0.9, 0.999), and then a cosine decay strategy was used to reduce the learning rate.
Preferably, the method for obtaining the preliminary reconstructed image by the IDFT of the completed sample fringes and the method for obtaining the preliminary OCT image by the IDFT of the complete sample interference fringes are respectively as follows: taking the completed sample stripes as input of IDFT (x) commands in matlab or python programs, and outputting the input after operation to obtain a preliminarily reconstructed image;
and taking the complete sample interference fringes as input of an IDFT (x) command in a matlab or python program, and obtaining a preliminary OCT image after operation.
Preferably, the method for calculating the image true value of the preliminary OCT image through a classical post-processing algorithm comprises the following steps:
the preliminary OCT image firstly takes a logarithmic function, and then obtains an image true value through thresholding and normalization.
Preferably, three groups of loss functions are constructed to evaluate the mask neural network after the current iteration, the enhancement neural network after the current iteration and the stripe completion neural network after the current iteration respectively, wherein the three groups of loss functions are specifically as follows:
wherein 
Representing a two-dimensional mask>
Represents +.>
Line and->
A column;
the above formula represents a statistical two-dimensional mask
The number of the middle pixel points is 1;
wherein 
Representing interference fringes, & lt & gt>
Representing interference fringes->
The%>
Line and->
A column;
the above formula is used to calculate the interference fringe after completion
And interference fringe truth->
Root mean square error of (a);
wherein 
Representing an image after image enhancement, < >>
Representing an enhanced image +.>
The%>
Line and->
A column;
the above formula is used to calculate an image after image enhancement
And image truth->
Average absolute error of (a);
wherein ,L mask film The data compression ratio is used for calculating the two-dimensional mask;
L stripe pattern The method is used for evaluating the effect of the stripe completion network after the current iteration;
L image processing apparatus The method is used for evaluating the effect of the image enhancement network after the current iteration;
and in the training stage of the mask neural network, the enhancement neural network and the stripe completion neural network, carrying out weighted summation on the three groups of loss functions, then counter-propagating gradients of the three groups of loss functions to carry out iterative solution on the mask neural network after the next iteration, the enhancement neural network after the next iteration and the stripe completion neural network after the next iteration until the similarity between the enhanced image and the true image is judged to reach the standard when the weighted summation value of the three groups of loss functions is no longer reduced, and obtaining a two-dimensional mask after the training is completed, the enhancement neural network after the training is completed and the stripe completion neural network after the training is completed, otherwise, continuing to train the mask neural network, the enhancement neural network and the stripe completion neural network.
The SS-OCT compression imaging system comprises a subsampled mask generating module, an interference fringe repairing module and an image enhancement module, wherein the subsampled mask generating module is used for sampling complete interference fringes acquired by the SS-OCT system to obtain undersampled interference fringes, and then the interference fringe repairing module and the image enhancement module are used for obtaining a target.
Compared with the prior art, the invention provides the SS-OCT compression imaging method for the deep learning and spectrum domain airspace combined sub-sampling, which has the following beneficial effects:
1. according to the invention, the mask neural network, the enhancement neural network and the stripe complement neural network are synchronously subjected to gradient descent update of parameters, so that the combined optimization of the two-dimensional mask and OCT image reconstruction is achieved, the problem of poor OCT image reconstruction effect caused by the adoption of an empirical fixed two-dimensional mask in the current OCT image sampling system is solved, and the OCT image reconstruction quality of the image sampling system is optimal under the same data compression ratio.
2. The invention counter propagates gradient by means of chained derivation, and simultaneously carries out gradient descent optimization updating on parameters of a mask neural network, an enhanced neural network and a stripe complement neural network, and the gradient descent optimization updating is carried out on the parameters of the mask neural network, the enhanced neural network and the stripe complement neural networkL Mask film 、L Stripe pattern AndL image processing apparatus After the three groups of loss functions are weighted and summed and converged, the best balance between the image reconstruction quality and the data compression rate can be obtained, and the problems that fixed empirical sub-sampling masks, such as a center mask and a random mask, cannot be optimized together with OCT image reconstruction, and the reconstructed OCT image quality is low are solved.
Drawings
FIG. 1 is a flow chart of the image sampling system construction of the present invention;
FIG. 2 is a schematic diagram showing the effect of reconstructing human cardiac muscle according to the present invention;
FIG. 3 is a schematic diagram of the reconstruction effect of a human finger according to the present invention;
FIG. 4 is a schematic diagram showing the onion skin reconstruction effect of the present invention;
FIG. 5 is a schematic diagram of the reconstruction effect of the comparative random mask and center mask of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As described in the background art, the present application provides an SS-OCT image compression method combining deep learning and spectral domain spatial domain joint sub-sampling in order to solve the above technical problems.
Embodiment one:
an SS-OCT compression imaging method for deep learning and spectral domain spatial domain joint sub-sampling, comprising:
step one: acquiring complete sample interference fringes and complete target detection interference fringes;
the method for acquiring the complete sample interference fringes and the complete target detection interference fringes comprises the following steps: the OCT system is used for collecting a sample object and an object to be detected to obtain a complete sample interference fringe and a complete target detection interference fringe, wherein the OCT system is used for converting coherent light into an electric signal through a photoelectric detector in the OCT system, and the electric signal is used for sampling through a data acquisition card to obtain the interference fringe.
Preferably, the sample interference fringes are preferably obtained by a dataset imaged by a commercial OCT system (Thorlabs Ganymede, newton, NJ), the dataset containing 3784 OCT images from 17 samples and corresponding interference fringes.
Step two: referring to fig. 1, an image sampling system is constructed and generated;
the construction and generation method of the image sampling system comprises the following steps:
s1: constructing a mask neural network, a stripe complement neural network and an enhancement neural network;
the mask neural network construction method comprises the following steps: the input of the mask neural network is random noise, and two groups of characteristic diagrams are output through four layers of up-sampling and four layers of down-sampling U-Net architectures, wherein each layer of up-sampling or down-sampling U-Net architecture comprises two layers of convolution layers;
the construction method of the stripe complement neural network comprises the following steps: the input of the stripe complement neural network is an undersampled interference stripe, and the complement sample stripe is obtained through the output of a four-layer downsampling and four-layer upsampling U-Net framework, wherein each layer of upsampling or downsampling U-Net framework comprises two layers of convolution layers;
the construction method of the enhanced neural network comprises the following steps: the input of the enhanced neural network is a preliminarily reconstructed image, and the enhanced OCT image is output through a four-layer downsampling and four-layer upsampling U-Net framework, wherein each layer of upsampling or downsampling framework comprises two layers of convolution layers.
S2: training a mask neural network, a stripe complement neural network and an enhancement neural network to respectively obtain a trained mask neural network, a trained stripe complement neural network and a trained enhancement neural network;
the method for training the mask neural network, the enhancement neural network and the stripe complement neural network comprises the following steps:
in the training framework, a batch size of 1 was used, an initial learning rate of 0.001, a network was trained for 30 cycles using an AdamW optimizer with momentum (0.9, 0.999), and then a cosine decay strategy was used to reduce the learning rate.
The concrete operation of the cosine decay strategy for reducing the learning rate is as follows: using the cosineAnneanlingLR function in Torch, the learning rate drops as a cosine function.
S21: inputting random noise into the mask neural network after the current nth iteration, and outputting and generating a two-dimensional mask; n is an integer, and the minimum value is 0;
when n=0, the corresponding mask neural network, stripe complement neural network and enhancement neural network are respectively the mask neural network, stripe complement neural network and enhancement neural network constructed in the S1;
generating a two-dimensional mask from the two sets of feature maps in the step S1 through GumbeSoftmax activation functions, wherein the GumbeSoftmax activation functions are as follows:
wherein D and
the output of the mask neural network and gummel Softmax activation function, respectively. We randomly select +.>
G is random noise of independent and identical samples in Gumbel (0, 1) distribution, τ is a value controlling Gumbel distribution density, p and c are pixel index and channel index, respectively,/">
The output of the gummel Softmax activation function, represented as channel c, pixel p; />
The output of the mask neural network is represented as channel c and pixel p; />
The output of the mask neural network is represented by the channel k and the pixel p; />
Output represented by random noise under channel k and pixel p, < >>
Represented as the output of random noise with channel c and pixel p.
S22: and multiplying the two-dimensional mask with the corresponding point of the complete sample interference fringe to obtain an undersampled fringe.
S23: and inputting the undersampled stripe into a stripe complement neural network after the current nth iteration to obtain a complement sample stripe.
S24: performing Inverse Discrete Fourier Transform (IDFT) on the completed sample stripes to obtain a primarily reconstructed image;
the method for obtaining the preliminarily reconstructed image by the completed sample stripes through IDFT (inverse discrete Fourier transform) comprises the following steps: and taking the completed sample stripes as input of IDFT (x) commands in matlab or python programs, and obtaining a preliminarily reconstructed image after operation.
S25: and inputting the preliminarily reconstructed image into an enhanced neural network after the current nth iteration, and outputting the enhanced image.
S26: the complete sample interference fringes and the IDFT (inverse discrete Fourier transform) are used for obtaining a preliminary OCT image, and the preliminary OCT image is calculated through a classical post-processing algorithm to obtain an image truth value;
the method for obtaining the preliminary OCT image by the complete sample interference fringes through IDFT (inverse discrete Fourier transform) comprises the following steps of: taking the complete sample interference fringes as input of IDFT (x) commands in matlab or python programs, and obtaining a preliminary OCT image after operation;
the method for calculating the image truth value of the preliminary OCT image through the classical post-processing algorithm comprises the following steps:
the preliminary OCT image firstly takes a logarithmic function, and then obtains an image true value through thresholding and normalization, namely: the preliminary OCT image is taken as input of a log10 (X) function in Matlab, then the value (marked as X) output by the logarithmic function is input into the Matlab, and the true value of the image is output according to the (X-3)/3 function.
S27: judging whether the similarity between the enhanced image and the true image meets the standard, if so, training the mask neural network after the current nth iteration, the stripe complement neural network after the current nth iteration and the enhanced neural network after the current nth iteration to obtain a trained mask neural network, a trained stripe complement neural network and a trained enhanced neural network, and completing the construction of an image sampling system, and entering step S10;
if the parameter does not reach the standard, updating parameters of the mask neural network after the current nth iteration, the stripe complement neural network after the current nth iteration and the enhancement neural network after the current nth iteration to obtain the mask neural network after the current n+1th iteration, the stripe complement neural network after the current n+1th iteration and the enhancement neural network after the current n+1th iteration, and returning to the step S2;
the method for updating the parameters of the enhanced neural network after the current nth iteration comprises the following steps: random gradient drop, namely: the network weights and bias terms are iteratively updated by computing gradients of the loss function over small batches of data.
The method for judging whether the similarity between the enhanced image and the true image meets the standard comprises the following steps:
three groups of loss functions are constructed to evaluate the mask neural network after the current nth iteration, the enhancement neural network after the current nth iteration and the stripe completion neural network after the current nth iteration respectively, wherein the three groups of loss functions are specifically as follows:
wherein 
Representing a two-dimensional mask>
Represents +.>
Line and->
A column;
the above formula represents a statistical two-dimensional mask
The number of the middle pixel points is 1;
wherein 
Representing interference fringes, & lt & gt>
Representing interference fringes->
The%>
Line and->
A column;
the above formula is used to calculate the interference fringe after completion
And interference fringe truth->
Root mean square error of (a);
wherein 
Representing an image after image enhancement, < >>
Representing an enhanced image +.>
The%>
Line and->
A column;
the above formula is used to calculate an image after image enhancement
And image truth->
Average absolute error of (a);
wherein ,L mask film The data compression ratio is used for calculating the two-dimensional mask;
L stripe pattern The method is used for evaluating the effect of the stripe completion network after the current nth iteration;
L image processing apparatus The method comprises the steps of evaluating the effect of an image enhancement network after the current nth iteration;
and in the training stage of the mask neural network, the enhancement neural network and the stripe completion neural network, carrying out weighted summation on the three groups of loss functions, and then counter-propagating gradients of the three groups of loss functions to carry out iterative solution on the mask neural network after the n+1st iteration, the enhancement neural network after the n+1st iteration and the stripe completion neural network after the n+1st iteration until the similarity of the enhanced image and the image true value reaches the standard when the weighted summation value of the three groups of loss functions is not reduced any more, so as to obtain a two-dimensional mask after the training is completed, the enhancement neural network after the training is completed and the stripe completion neural network after the training is completed, otherwise, continuing training the mask neural network, the enhancement neural network and the stripe completion neural network. The calculation mode and structure of the loss function are the prior art, and the invention only performs transfer.
S3: an image sampling system is generated that includes the current two-dimensional mask, the trained stripe completion neural network, the trained enhancement neural network, and the IDFT.
Step three: and inputting the complete target detection interference fringes into the image sampling system to obtain a target image.
The invention counter propagates gradient by means of chained derivation, and simultaneously carries out gradient descent optimization updating on parameters of a mask neural network, an enhanced neural network and a stripe complement neural network, and the gradient descent optimization updating is carried out on the parameters of the mask neural network, the enhanced neural network and the stripe complement neural networkL Mask film 、L Stripe pattern AndL image processing apparatus After the three groups of loss functions are weighted and summed and converged, the best balance between the image reconstruction quality and the data compression rate can be obtained, and the problems that fixed empirical sub-sampling masks, such as a center mask and a random mask, cannot be optimized together with OCT image reconstruction, and the reconstructed OCT image quality is low are solved.
Embodiment two:
referring to fig. 2-5, an SS-OCT compression imaging system with deep learning and spectrum domain spatial domain combined sub-sampling uses the SS-OCT image compression method, which includes a sub-sampling mask generation module, an interference fringe repairing module and an image enhancement module, and the sub-sampling mask module samples the complete interference fringe acquired by the SS-OCT system to obtain an undersampled fringe, and then the interference fringe repairing module and the image enhancement module obtain a target image.
The sub-sampling mask generation module generates a sub-sampling mask through input noise, and the sub-sampling mask samples data acquired by the SS-OCT system to form undersampled stripes; then, repairing the undersampled stripe by an interference stripe repairing module to obtain a completed sample stripe; and the image enhancement module performs image enhancement operation on the completed sample stripes to obtain a target image.
Further, the sub-sampling mask generation module comprises a U-Net architecture with 1 channel input and 2 channels output and a Gumbel Softmax activation function. The U-Net architecture comprises a four-layer downsampling module, a four-layer upsampling module and a jump connection. The Gumbel Softmax activation function encodes the output of the U-Net architecture, generating a (0, 1) sub-sampling mask.
The interference fringe repairing module comprises: the input is 1 channel, the output is 1 channel U-Net architecture and IDFT (inverse discrete Fourier transform). The U-Net architecture comprises a four-layer downsampling module, a four-layer upsampling module and a jump connection. The undersampled streaks are first complemented by the U-Net architecture to obtain complemented sample streaks, and then the complemented sample streaks are converted into a preliminarily reconstructed image by IDFT.
The image enhancement module comprises: the input is a U-Net architecture with 2 channels and 1 channel output. And carrying out image enhancement on the preliminarily reconstructed image through a U-Net architecture to obtain an enhanced image (namely, a target image).
The image compression system adopts an L1 norm of a difference between a calculated target image and a true value image, an L2 norm of a difference between a repair stripe (namely a completed sample stripe) and an original stripe (namely a complete sample interference stripe) and a data compression ratio (namely a total 1 ratio in a formed two-dimensional mask), and specifically comprises the following steps:
wherein 
For the total loss function +.>
An L1 norm that is the difference between the target image and the truth image;
is->
Weights of (2); />
An L2 norm for repairing the difference between the stripe and the original stripe; />
Is that
Weights of (2); />
Is the total number of 1's in the two-dimensional mask;
further, a sub-sampling mask generating module is adopted to sample the complete stripe acquired by the SS-OCT through a sub-sampling mask generated by input noise, and then an enhanced image (namely a target image) is obtained through a stripe complementing module and an image enhancing module, specifically:
a1, for the complete sample interference fringes acquired by SS-OCT, adjusting the interference fringes to an image with 2048 x 992 resolution through zero padding or clipping;
a2, inputting noise into a generator in the sub-sampling mask generation module, and generating a sub-sampling mask (namely a two-dimensional mask) with only 0 and 1 through a Gumbel Softmax activation function;
a3, multiplying the sub-sampling mask generated in the step a2 with the corresponding points of the complete sample interference fringes to obtain undersampled fringes;
a4, obtaining a completed sample stripe by a stripe repair module for the undersampled stripe obtained in the step a 3;
a5, obtaining a preliminarily reconstructed image (namely an preliminarily reconstructed OCT image) of the completed sample stripe obtained in the step a4 through IDFT operation;
a6, obtaining an enhanced image (namely a target image) of the primarily reconstructed image obtained in the step a5 through an image enhancement module;
the subsampling mask generating module, the interference fringe repairing module and the image enhancement module are all U-Net architecture networks, all the architecture networks are provided with four downsampling (and corresponding upsampling) blocks, and the initial characteristics of the input layer are set to be 32 characteristics. The downsampling block uses a leaklylrelu (negative slope 0.2) activation function and the upsampling block uses a ReLU activation function. All blocks were normalized using a batch, with two convolutional layers for each block. The sub-sampling mask generation module has 1 input channel and 2 output channels. The interference fringe repairing module is provided with 1 input channel and 1 output channel. The image enhancement module has 2 input channels and 1 output channel.
In the training framework, the batch size used was 1, the initial learning rate was 0.001, the network was trained for 30 cycles using an AdamW optimizer with momentum (0.9,0.999), and then the cosine decay strategy was used to reduce the learning rate. All experiments were trained and tested using NVIDIA GeForce RTX 3090 GPU card.
Further, two human coronary arteries, two human fingers and one onion collected by the same device were used as test sets to further illustrate the generalization of the proposed technique.
Case one:
the performance of the present invention was evaluated using peak signal to noise ratio (PSNR) and Structural Similarity Index (SSIM), and experiments were performed on human coronary arteries, human fingers and onions, respectively, and the experimental results are shown in table 1.
TABLE 1
Obviously, the image quality steadily decreases with the increase of the compression rate; the same trend was observed when the test set was deviated from the coronary artery sample to the finger and onion. However, even if a large amount of original data is lost, the original image can be restored; when only 1.6% of the data is used, a PSNR of 24.2 dB can be achieved. Three exemplary reconstructed OCT images (one coronary artery, one finger, and one onion) obtained by using 1.6%, 3.6%, 10%, 25%, and 50% of the raw data and the corresponding truth images are shown in fig. 2, fig. 3, fig. 4, respectively. Most texture information is well preserved. Wherein DCR is the data compression rate.
Case two:
in this case, compared to case two, a comparison study is performed on the human coronary dataset by replacing the learnable mask with other empirically fixed mask patterns, such as random spectrum sub-sampling (referred to as "random mask") and center truncated spectrum (referred to as "center mask"). It should be noted that the entire network is retrained and trimmed for each different mask pattern. Fig. 5 shows the corresponding results for different DCRs.
The proposed learner mask performs best in all mask patterns for all different DCRs. In particular for very high DCR (1%), only the proposed method is able to reconstruct images with a PSNR exceeding 21 dB. In contrast, the center mask performs relatively well at low DCR (50%), with the corresponding PSNR almost 1.8 dB higher than the random mask, 2.2 dB lower than the present invention. However, as DCR increases (e.g., 1% and 5%), its performance begins to perform poorly, streak artifacts appear; on the other hand, the reconstruction result obtained by using the random mask is only degraded with the increase of DCR.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made hereto without departing from the spirit and principles of the present invention.
Claims (8)
1. The SS-OCT image compression method for the deep learning and spectral domain airspace combined sub-sampling is characterized by comprising the following steps of:
the method comprises the following steps:
step one, constructing and training each neural network;
constructing a mask neural network:
the input of the mask neural network is random noise, two sets of feature graphs are output through a U-Net architecture of four layers of up-sampling and four layers of down-sampling, wherein each layer of up-sampling or down-sampling comprises two layers of convolution layers, and the two last sets of feature graphs generate a two-dimensional mask through Gumbel Softmax activation functions;
wherein Gumbel Softmax activation function is:
wherein D and
the output of the mask neural network and Gumbel Softmax activation function are respectively; random selection->
As a two-dimensional mask; g is random noise of independent and identical samples in Gumbel (0, 1) distribution, τ is a value controlling Gumbel distribution density, p and c are pixel index and channel index, respectively, # and (x) are the values of the Gumbel distribution density>
The output of the gummel Softmax activation function, represented as channel c, pixel p; />
The output of the mask neural network is represented as channel c and pixel p;
the output of the mask neural network is represented by the channel k and the pixel p; />
Represented by random noise under channel k and pixel pOutput (I)>
An output representing random noise with channel c and pixel p;
the method for constructing the stripe complement neural network comprises the following steps:
the input of the stripe complement neural network is undersampled stripe, and the complement sample stripe is obtained through the output of the U-Net architecture of four layers of downsampling and four layers of upsampling, wherein each layer of upsampling or downsampling comprises two layers of convolution layers;
the method for constructing the enhanced neural network comprises the following steps:
the input of the enhanced neural network is a preliminarily reconstructed image, and the enhanced OCT image is obtained through the output of a U-Net architecture of four layers of downsampling and four layers of upsampling, wherein each layer of upsampling or downsampling comprises two layers of convolution layers;
training the mask neural network, the stripe complement neural network and the enhancement neural network until convergence to obtain a trained mask neural network, a trained stripe complement neural network and a trained enhancement neural network;
the convergence time is the time when the similarity between the enhanced image and the true image reaches the standard;
the enhanced image is obtained by the following steps: inputting random noise into the mask neural network after the current iteration, and outputting and generating a two-dimensional mask; multiplying the two-dimensional mask with the corresponding point of the complete sample interference fringe to obtain an undersampled fringe; inputting the undersampled stripe into a stripe complement neural network after the current iteration to obtain a complement sample stripe; obtaining a preliminarily reconstructed image by carrying out IDFT on the completed sample stripes; inputting the preliminarily reconstructed image into an enhanced neural network after the current iteration, and outputting to obtain an enhanced image;
generating an image sampling system, wherein the image sampling system comprises a two-dimensional mask generated by outputting a mask neural network after training, a stripe complement neural network after training, an enhancement neural network after training and IDFT;
step three: and inputting the complete target detection interference fringes into the image sampling system to obtain a target image.
2. The SS-OCT image compression method of joint sub-sampling of deep learning and spectral domain spatial domain of claim 1, wherein: the method for acquiring the complete sample interference fringes and the complete target detection interference fringes comprises the following steps: and acquiring the sample object and the object to be detected through the OCT system to obtain complete sample interference fringes and complete target detection interference fringes.
3. The SS-OCT image compression method of joint sub-sampling of deep learning and spectral domain spatial domain of claim 1, wherein: the image truth value is obtained by the following steps: and obtaining a preliminary OCT image by the integral sample interference fringes and IDFT, and obtaining an image truth value by calculating the preliminary OCT image through a classical post-processing algorithm.
4. The SS-OCT image compression method of joint sub-sampling of deep learning and spectral domain spatial domain of claim 1, wherein: the method for training the mask neural network, the enhanced neural network and the stripe complement neural network comprises the following steps:
in the training framework, a batch size of 1 was used, an initial learning rate of 0.001, a network was trained for 30 cycles using an AdamW optimizer with momentum (0.9, 0.999), and then a cosine decay strategy was used to reduce the learning rate.
5. The SS-OCT image compression method of joint sub-sampling of deep learning and spectral domain spatial domain of claim 1, wherein: the method for obtaining the preliminarily reconstructed image by the completed sample fringes through IDFT and the method for obtaining the preliminarily OCT image by the complete sample interference fringes through IDFT are respectively as follows: taking the completed sample stripes as input of IDFT (x) commands in matlab or python programs, and outputting the input after operation to obtain a preliminarily reconstructed image;
and taking the complete sample interference fringes as input of an IDFT (x) command in a matlab or python program, and obtaining a preliminary OCT image after operation.
6. The SS-OCT image compression method of joint sub-sampling of deep learning and spectral domain spatial domain of claim 1, wherein: the method for calculating the image truth value of the preliminary OCT image through the classical post-processing algorithm comprises the following steps:
the preliminary OCT image firstly takes a logarithmic function, and then obtains an image true value through thresholding and normalization.
7. The SS-OCT image compression method of claim 6, wherein the SS-OCT image is compressed using a combination of deep learning and spectral domain spatial sub-sampling, wherein: three groups of loss functions are constructed to evaluate the mask neural network after the current iteration, the enhancement neural network after the current iteration and the stripe completion neural network after the current iteration respectively, wherein the three groups of loss functions are specifically as follows:
wherein 
Representing a two-dimensional mask>
Representing a two-dimensional mask->
The%>
Line and->
A column;
the above formula represents a statistical two-dimensional mask
The number of the middle pixel points is 1;
wherein 
Representing interference fringes, & lt & gt>
Representing interference fringes->
The%>
Line and->
A column;
the above formula is used to calculate the interference fringe after completion
And interference fringe truth->
Root mean square error of (a);
wherein 
Representing an image after image enhancement, < >>
Representing an enhanced image +.>
The%>
Line and->
A column;
the above formula is used to calculate an image after image enhancement
And image truth->
Average absolute error of (a);
wherein ,L mask film The data compression ratio is used for calculating the two-dimensional mask;
L stripe pattern The method is used for evaluating the effect of the stripe completion network after the current iteration;
L image processing apparatus The method is used for evaluating the effect of the image enhancement network after the current iteration;
and in the training stage of the mask neural network, the enhancement neural network and the stripe completion neural network, carrying out weighted summation on the three groups of loss functions, then counter-propagating gradients of the three groups of loss functions to carry out iterative solution on the mask neural network after the next iteration, the enhancement neural network after the next iteration and the stripe completion neural network after the next iteration until the similarity between the enhanced image and the true image is judged to reach the standard when the weighted summation value of the three groups of loss functions is no longer reduced, and obtaining a two-dimensional mask after the training is completed, the enhancement neural network after the training is completed and the stripe completion neural network after the training is completed, otherwise, continuing to train the mask neural network, the enhancement neural network and the stripe completion neural network.
8. An SS-OCT compression imaging system with deep learning and spectral domain spatial domain joint sub-sampling, using the SS-OCT image compression method of any one of claims 1-7, characterized in that: the method comprises a sub-sampling mask generation module, an interference fringe repair module and an image enhancement module, wherein the sub-sampling mask module is used for sampling complete interference fringes acquired by an SS-OCT system to obtain undersampled fringes, and then the interference fringe repair module and the image enhancement module are used for obtaining a target image.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310264537.8A CN115984406B (en) | 2023-03-20 | 2023-03-20 | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling |
| PCT/CN2024/088631 WO2024193719A1 (en) | 2023-03-20 | 2024-04-18 | Ss-oct compressive imaging method based on deep learning and spectral domain and spatial domain joint sub-sampling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310264537.8A CN115984406B (en) | 2023-03-20 | 2023-03-20 | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115984406A CN115984406A (en) | 2023-04-18 |
| CN115984406B true CN115984406B (en) | 2023-06-20 |
Family
ID=85963478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310264537.8A Active CN115984406B (en) | 2023-03-20 | 2023-03-20 | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115984406B (en) |
| WO (1) | WO2024193719A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984406B (en) * | 2023-03-20 | 2023-06-20 | 始终(无锡)医疗科技有限公司 | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113674376A (en) * | 2021-08-23 | 2021-11-19 | 合肥工业大学 | Ultrasonic plane wave composite imaging method based on deep learning |
| CN114331911A (en) * | 2022-01-05 | 2022-04-12 | 北京理工大学重庆创新中心 | Fourier laminated microscopic image denoising method based on convolutional neural network |
| CN114820849A (en) * | 2022-04-19 | 2022-07-29 | 浙江大学 | Magnetic resonance CEST image reconstruction method, device and equipment based on deep learning |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4389032B2 (en) * | 2007-01-18 | 2009-12-24 | 国立大学法人 筑波大学 | Optical coherence tomography image processing device |
| US20210369195A1 (en) * | 2018-04-26 | 2021-12-02 | Voxeleron, LLC | Method and system for disease analysis and interpretation |
| CN109829880A (en) * | 2018-12-07 | 2019-05-31 | 清影医疗科技（深圳）有限公司 | A kind of CT image detecting method based on deep learning, device and control equipment |
| US20220058803A1 (en) * | 2019-02-14 | 2022-02-24 | Carl Zeiss Meditec Ag | System for oct image translation, ophthalmic image denoising, and neural network therefor |
| JP2023523246A (en) * | 2020-04-29 | 2023-06-02 | ノバルティス アーゲー | Computer-implemented system and method for assessing disease or condition activity level in a patient's eye |
| WO2021220008A1 (en) * | 2020-04-29 | 2021-11-04 | Deep Render Ltd | Image compression and decoding, video compression and decoding: methods and systems |
| WO2022008986A1 (en) * | 2020-07-09 | 2022-01-13 | Nidek Technologies S.R.L. | A computer-implemented method for executing semantic segmentation of rgb digital images |
| CN111798535B (en) * | 2020-09-09 | 2020-12-01 | 南京安科医疗科技有限公司 | CT image enhancement display method and computer readable storage medium |
| CN113658115B (en) * | 2021-07-30 | 2023-02-14 | 华南理工大学 | Image anomaly detection method based on depth convolution generation countermeasure network |
| CN114723840A (en) * | 2022-03-09 | 2022-07-08 | 天津大学 | Slice-adaptive-oriented active undersampling method for magnetic resonance imaging |
| CN115797231A (en) * | 2022-12-05 | 2023-03-14 | 上海交通大学 | Real-time hologram generation method based on neural network of Fourier inspiration |
| CN115984406B (en) * | 2023-03-20 | 2023-06-20 | 始终(无锡)医疗科技有限公司 | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling |
-
2023
- 2023-03-20 CN CN202310264537.8A patent/CN115984406B/en active Active
-
2024
- 2024-04-18 WO PCT/CN2024/088631 patent/WO2024193719A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113674376A (en) * | 2021-08-23 | 2021-11-19 | 合肥工业大学 | Ultrasonic plane wave composite imaging method based on deep learning |
| CN114331911A (en) * | 2022-01-05 | 2022-04-12 | 北京理工大学重庆创新中心 | Fourier laminated microscopic image denoising method based on convolutional neural network |
| CN114820849A (en) * | 2022-04-19 | 2022-07-29 | 浙江大学 | Magnetic resonance CEST image reconstruction method, device and equipment based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115984406A (en) | 2023-04-18 |
| WO2024193719A1 (en) | 2024-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109035142B (en) | Satellite image super-resolution method combining countermeasure network with aerial image prior | |
| CN111127320B (en) | Deep learning-based photoacoustic image super-resolution reconstruction method and device | |
| CN113344799A (en) | System and method for reducing colored noise in medical images using deep neural networks | |
| Deora et al. | Structure preserving compressive sensing MRI reconstruction using generative adversarial networks | |
| CN110880196B (en) | Tumor photoacoustic image rapid reconstruction method and device based on deep learning | |
| CN113376600B (en) | Pedestrian radar echo denoising method based on RSDNet | |
| CN114648048B (en) | Electrocardiosignal noise reduction method based on variational self-coding and PixelCNN model | |
| CN115984406B (en) | SS-OCT compression imaging method for deep learning and spectral domain airspace combined sub-sampling | |
| CN116563110A (en) | Blind image super-resolution reconstruction method based on Bicubic downsampling image space alignment | |
| CN115496659A (en) | Three-dimensional CT image reconstruction method and device based on single projection data | |
| CN106023116B (en) | A kind of compressed sensing image rebuilding method and device based on block Weighted Constraint | |
| CN115375785A (en) | Magnetic resonance image reconstruction method and device based on artificial neural network | |
| CN117094897B (en) | Super-resolution phase gradient estimation method for phase contrast optical coherence elastography | |
| CN111815692A (en) | Method, system and storage medium for generating artifact-free data and artifact-containing data | |
| CN114414050B (en) | Self-adaptive Fourier calculation correlation imaging method and system | |
| CN117115452A (en) | Controllable medical ultrasonic image denoising method, system and computer storage medium | |
| CN112634385B (en) | Rapid magnetic resonance imaging method based on deep Laplace network | |
| CN117974468A (en) | Multi-mode medical image fusion method for global and local feature interaction parallelism | |
| CN113744132B (en) | MR image depth network super-resolution method based on multiple optimization | |
| CN116630154A (en) | Deconvolution super-resolution reconstruction method and device for optical coherence tomography | |
| US20230135995A1 (en) | Multi-slice mri data processing using deep learning techniques | |
| CN115526792A (en) | Point spread function prior-based coding imaging reconstruction method | |
| He et al. | Learning priors in high-frequency domain for inverse imaging reconstruction | |
| CN115100069A (en) | Ultrasonic image reconstruction method, system, device and medium | |
| CN118014016B (en) | OCT image reconstruction method based on antagonism convolution neural network and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |