CN113870331B - Chest CT and X-ray real-time registration algorithm based on deep learning - Google Patents
Chest CT and X-ray real-time registration algorithm based on deep learning Download PDFInfo
- Publication number
- CN113870331B CN113870331B CN202111167612.6A CN202111167612A CN113870331B CN 113870331 B CN113870331 B CN 113870331B CN 202111167612 A CN202111167612 A CN 202111167612A CN 113870331 B CN113870331 B CN 113870331B
- Authority
- CN
- China
- Prior art keywords
- points
- coordinate system
- feature point
- pose
- ray
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013135 deep learning Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 claims description 67
- 238000001514 detection method Methods 0.000 claims description 59
- 238000012549 training Methods 0.000 claims description 28
- 238000003384 imaging method Methods 0.000 claims description 21
- 239000011159 matrix material Substances 0.000 claims description 18
- 230000008447 perception Effects 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 17
- 238000013519 translation Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 16
- 238000010276 construction Methods 0.000 claims description 16
- 239000013598 vector Substances 0.000 claims description 15
- 238000013527 convolutional neural network Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 claims description 8
- 238000011976 chest X-ray Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 238000011156 evaluation Methods 0.000 claims description 2
- 241000251539 Vertebrata <Metazoa> Species 0.000 abstract description 2
- 230000029058 respiratory gaseous exchange Effects 0.000 abstract description 2
- 230000006872 improvement Effects 0.000 description 7
- 238000001356 surgical procedure Methods 0.000 description 6
- 230000009467 reduction Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000002591 computed tomography Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000241 respiratory effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10116—X-ray image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
- G06T2207/30012—Spine; Backbone
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a chest CT and X-ray real-time registration algorithm based on deep learning, which is used for realizing accurate and rapid initialization registration. The invention uses the spine center which is not affected by respiration basically in operation as the anatomical feature point, and the registration of CT and X-ray is completed by the assistance of feature point matching. In order to adapt to the characteristic that the central points of the vertebrates are in the spatial coplanarity, the invention converts the problem of 6-degree-of-freedom pose estimation into 4-degree-of-freedom pose estimation by taking the conventional pose of C-shaped arm scanning as a priori, the number of required feature point matching is less, and the problem of unavoidable feature point positioning errors in operation is solved.
Description
Technical Field
The invention belongs to the crossing field of medicine and computer science, and particularly relates to a chest CT and X-ray real-time registration algorithm based on deep learning.
Background
In the image-guided minimally invasive surgery, a two-dimensional X-ray image shot by a C-shaped arm X-ray machine can irradiate the area where a focus is located in real time, guide a doctor to find the focus and execute surgery operation, but the focus is generally close to human muscle tissue in density, the focus is not easy to distinguish on the X-ray image, and other tissues and organs are overlapped and shielded, so that the doctor is difficult to accurately judge the focus position on the X-ray image by naked eyes, and the surgery difficulty is increased. The focus positioning means commonly used at present is that three-dimensional CT scanning is firstly carried out on a patient before an operation, a doctor can accurately mark the focus position in CT in preoperative planning due to the fact that CT images have three-dimensional information and high precision, and in the operation process, a registration algorithm is used for aligning the CT images with two-dimensional X-ray images acquired in the operation, so that the focus position marked in CT is projected onto the X-ray images, enhancement of the X-ray images is achieved, and the doctor is guided to accurately position the focus. However, in order to ensure the accuracy of registration, the conventional registration algorithm often needs to be optimized for a long time and has a smaller convergence domain, so that the requirements of automatic and real-time registration in the operation process cannot be met. In order to ensure the accuracy and the high efficiency of the registration, the registration is executed in two stages, wherein the first stage is global initialization, CT is transformed from an initial posture to a position close to a target posture and enters a convergence domain of a traditional algorithm, the second stage is local optimization, and the initialized posture is finely adjusted by the traditional algorithm. Global initialization typically requires the establishment of associations between identical anatomical feature points in CT and X-rays, thereby calculating a closed-form solution for pose transformation using algorithms that minimize re-projection errors, such as PnP. The detection of the anatomical feature points is usually completed by using a deep learning algorithm, and compared with the traditional manual feature design method, the deep learning method greatly improves the accuracy and the detection rate of the anatomical feature. However, even so, the existing deep learning method cannot ensure that all anatomical feature points are accurately detected, and especially for an X-ray image with poor imaging quality, erroneous anatomical feature association is caused, so that a large initialization error is caused. In recent years, some studies have proposed giving different weights to different anatomical feature points, but these offline weights cannot adapt to the scene changed during surgery, and registration failure is likely to occur.
Disclosure of Invention
Aiming at the advantages and disadvantages of the algorithm and the existing problems, the invention provides a chest CT and X-ray real-time registration algorithm based on deep learning, which is used for realizing accurate and rapid initialization registration. The invention uses the spine center which is not affected by respiration basically in operation as the anatomical feature point, and the registration of CT and X-ray is completed by the assistance of feature point matching. In order to adapt to the characteristic that the central points of the vertebrates are in the spatial coplanarity, the invention converts the problem of 6-degree-of-freedom pose estimation into 4-degree-of-freedom pose estimation by taking the conventional pose of C-shaped arm scanning as a priori, the number of required feature point matching is less, the number of candidate poses is increased, and the robustness of registration is effectively improved. Aiming at the problem of unavoidable feature point positioning errors in operations, the invention provides a robust pose fusion method with error perception, and the candidate poses calculated according to feature matching are input, so that the error feature points can be estimated and restrained simultaneously, and then correct candidate poses are fused, thereby realizing robust initialization registration and improving the accuracy and efficiency of focus positioning in chest surgery.
The invention is realized by the following technical scheme:
The invention discloses a chest CT and X-ray real-time registration method based on deep learning, which comprises the following steps:
1) Acquiring a three-dimensional chest CT flat scanning image of a patient before an operation, positioning a spine center point in CT by using a 3D feature point detection unit, checking a positioning result of the 3D feature point, and ensuring the precision of the 3D feature point;
2) Acquiring a two-dimensional chest X-ray image of a patient in an operation, positioning a spine center point in X-rays by using a 2D feature point detection unit, and establishing a series of 2D-3D feature point correlations by using 3D feature points positioned in advance;
3) Inputting the established association points and the shooting gestures of the C-shaped arm into a gesture solving unit, and solving a 6-degree-of-freedom candidate gesture by using association of every two feature points;
4) Inputting all the solved candidate poses into a robust estimation unit with error perception, estimating feature points positioned in error, and aggregating all the candidate poses to obtain robust and accurate poses.
As a further improvement, the 3D feature point detection unit according to the present invention includes the following construction processes:
constructing a chest flat scanning CT data set marked with a spine central point;
Constructing a 3D feature point detection model based on a deep convolutional neural network, training by a network training method, outputting a thermodynamic diagram with N channels, the size of which is the same as that of an original image, by the network, wherein channel serial numbers represent the types of feature points, namely N vertebrae forming a human spine, and the position with the maximum thermodynamic diagram response in each channel is the estimated position of a spine center point;
Based on the trained 3D characteristic point detection model, spine center point positioning is carried out on chest CT flat scanning images of patients before operation, and in order to ensure positioning accuracy of the 3D characteristic points, positioning results are checked and corrected.
As a further improvement, the data set construction method of the present invention includes the steps of:
Acquiring chest flat scanning CT image data, marking all spine center points in an image view, and respectively assigning numbers of 0-N to N vertebrae of a human body;
collecting a series of intraoperative X-ray images or CT locating slice images of a corresponding patient, and performing long-time offline optimization calculation by using a traditional registration method, so as to obtain an accurate registration result, wherein the accurate registration result is used as a true value of registration for network training;
and (3) projecting the 3D spine central point marked in advance in CT to an X-ray plane by using the registration results, thereby finishing marking the spine central point in X-ray.
As a further improvement, the network training method of the present invention includes the steps of:
training a 3D feature point detection model based on the constructed CT data set marked with the spine center point;
Based on the established X-ray data set marked with the spine center point, a 2D characteristic point detection model is pre-trained, 2D-3D characteristic point matching is established by utilizing CT and X-rays, registration results are estimated by using a pose calculation unit and a robust estimation unit with error perception, end-to-end training is carried out, the estimated poses are supervised by using registration truth poses, and the 2D characteristic point detection model is further trained.
As a further improvement, the 2D feature point detection unit according to the present invention includes the following construction processes:
constructing a chest X-ray data set marked with a spine central point;
2D feature point detection models are built and trained based on the deep convolutional neural network, the basic framework of the 2D feature point detection models is consistent with the 3D feature point detection models, and the data transmission mode of the neural network is changed from 3D to 2D;
And (3) based on the trained 2D feature point detection model, performing feature point positioning on the X-ray image acquired in real time in an operation.
As a further improvement, the pose solving unit of the present invention includes the following construction processes:
Based on the results of the 3D and 2D feature point detection units, a series of 2D-3D feature point matches are established, the gesture photographed by the C-shaped arm is utilized to provide two priori rotation angles beta and gamma, and two feature point matches are given, wherein the 3D feature points under the CT coordinate system are expressed as AndThe corresponding 2D feature points in the X-ray image coordinate system are represented as p 1 and p 2, and the projection equations of the 2D and 3D points in the C-arm coordinate system are as follows
Where i= {1,2}, the index representing the matching point,Representing the z-coordinate of the 3D point in the C-arm coordinate system, K represents the internal reference of the C-arm imaging system, and the conversion relation from the CT coordinate system to the C-arm coordinate system is expressed as
Wherein the method comprises the steps ofRepresenting the rotation and translation matrices, which are the solving targets for the registration task, is simplified as T, since the rotation angles beta, gamma are known,Can be decomposed by the mode of 'xyz' compliance
In the camera coordinate system, two unit vectors q 1、q2 are defined, which respectively represent direction vectors from the origin to two 3D points, and depths along the z-axis from the origin to the two 3D points are respectively denoted as μ 1、μ2, resulting in the following equation
Wherein the method comprises the steps ofRepresenting the coordinates of a 3D point in an intermediate coordinate system, i being 1 or 2, representing the numbering of two 3D points, the z-axis of the intermediate coordinate system being aligned with the z-axis of the C-arm imaging coordinate system, in order to calculate the rotation angle α and the translation matrixFirstly, an unknown depth mu 1、μ2 needs to be calculated, and the calculation is carried out by using the constraint provided by two 3D points, wherein the first constraint is that the distance between the two 3D points in an intermediate coordinate system is the same as the distance between the 3D points in a C-shaped arm imaging coordinate system
Since the rotation angle α is rotated only around the z-axis, the projection of the two 3D points on the z-axis of the intermediate coordinate system must be the same as in the C-arm imaging coordinate system, the second constraint being expressed as
According to the above two equations, a unitary quadratic equation is established with respect to μ 2, whereby a is calculated according to
Wherein the method comprises the steps ofA unit vector representing a vector x is provided,Calculated according to mu i and q i, the translation matrix is calculated according to the following formula
As a further improvement, the robust estimation unit with error perception according to the invention comprises the following construction process:
based on the pose solving unit, traversing all possible feature point matching combinations, and calculating a candidate pose set Wherein N is the number of spines appearing in CT and X-ray fields at the same time, after the construction of the candidate pose set is completed, weight is allocated to each candidate pose, and the candidate pose is determined to be the correct pose or the outlier pose, and for each candidate pose3D feature points in CT coordinate systemWill be transformed into the C-arm imaging coordinate system and then projected into the 2D image coordinate system according to the projection equation to obtain a series of 2D points2D feature points using 2D-SCN predictionCalculating the distance between 2D feature points:
for the following The formed distance set isEvaluation by way of a plausible calculation of the confidence levelAs confidence of correct pose, a valid feature point mask is calculatedFor selecting those feature points whose distance is less than a threshold value τ
Since the distance error is inversely proportional to the confidence, the confidence can be expressed as
If the denominator is 0, then c i will be set to 0, i.e. the candidate pose will be ignored, and the weight corresponding to each candidate pose can be expressed as
Normalization was performed using softmax, where λ is the temperature coefficient, and for all candidate poses, the set of weights is expressed asFinal robust pose estimation is achieved using weighted averaging
Wherein the method comprises the steps ofFrom the following componentsIt is deduced that since two a priori rotation angles β, γ are known, the rotation matrix can be estimated from α, and likewise the translation matrix can be obtained by weighted averaging as follows
Through the operation, the robust pose estimation unit with error perception is tiny, and end-to-end training is realized.
Compared with the prior art, the invention has the following beneficial effects:
The invention discloses a chest CT and X-ray real-time registration algorithm based on deep learning, which aims at the problem that a spine central point is coplanar in a three-dimensional space when a pose solving unit is constructed. Aiming at the problem that the positioning of the feature points of the existing initialization registration method is inaccurate, the registration accuracy is affected, by constructing a robust estimation unit with error perception, and utilizing a re-projection error and a micro pose aggregation method, the error positioning feature points are estimated and the correct candidate poses are fused, so that the registration accuracy and the registration robustness are greatly improved. Because the algorithm has no iterative process in the inference process, all results can be solved analytically, and the registration efficiency is high, the method can be applied to real-time positioning scenes in operation.
At present, most of the existing CT and X-ray registration algorithms stay in local registration, and the registration result is greatly influenced by an initial value. The invention can thus be used to initialize any local registration algorithm for further performance improvement.
In addition, the dimension of the pose estimation with 6 degrees of freedom is reduced through two priori rotation angles, so that degradation caused by coplanarity of 3D feature points can be effectively avoided, in addition, the pose solving process after dimension reduction only needs matching of 23D feature points and 2D feature points, compared with the matching of 3 required before dimension reduction, the success rate of registration can be improved under the condition that error exists in feature point positioning after dimension reduction, and the robustness of a system is improved. For the prior angle, the invention can resist the prior angle error of not less than 15 degrees under the condition that the success rate is more than 70 percent.
Compared with the traditional registration algorithm and the existing deep learning-based algorithm, the method has the advantages that the characteristic point error positioning condition is specifically designed, the error positioning characteristic point is estimated by using the robust estimation unit with error perception, and the correct pose is fused, so that the registration method can obtain higher registration precision and higher success rate of registration.
All modules of the invention are tiny, so the algorithm framework of the invention can carry out end-to-end training, not only carry out supervision through the true value of feature point positioning, but also carry out supervision according to the true value of the pose, so that the network can be further converged to global optimum, and higher registration precision and robustness are obtained.
In chest surgery, chest CT and X-ray registration is always a problem with great difficulty, because the respiratory action of a patient can bring about the change of X-ray images, and the traditional registration algorithm cannot realize real-time registration, so that the timeliness of registration results is poor.
Drawings
FIG. 1 is a flow chart of a chest CT and X-ray registration algorithm based on deep learning;
Fig. 2 is a schematic diagram of the coordinate system of chest CT and X-ray registration of the present invention.
Detailed Description
The invention discloses a chest CT and X-ray real-time registration algorithm based on deep learning, which is used for acquiring a three-dimensional chest CT flat scanning image of a patient before an operation, realizing the positioning of a spine central point in the CT by using a 3D characteristic point detection unit, checking the positioning result of the 3D characteristic point by a doctor with rich experience, ensuring the precision of the 3D characteristic point, acquiring a two-dimensional chest X-ray image of the patient during the operation, realizing the positioning of the spine central point in X-rays by using a 2D characteristic point detection unit, and establishing a series of 2D-3D characteristic point correlations by using the 3D characteristic point positioned in advance. The established association points and the shooting gestures of the C-shaped arm are input into a gesture solving unit, a 6-degree-of-freedom candidate gesture can be solved by using association of every two feature points, then all solved candidate gestures are input into a robust estimation unit with error perception, and all candidate gestures are aggregated to obtain a robust and accurate gesture.
The construction process of the 3D feature point detection unit is as follows:
constructing a chest flat scanning CT data set marked with a spine central point;
A3D characteristic point detection model is built based on a deep convolutional neural network and trained, the network outputs a thermodynamic diagram with 26 channels as large as an original image, channel serial numbers represent the types of characteristic points, namely the serial numbers C1 to S1 of 26 vertebrae, and the position with the maximum thermodynamic diagram response in each channel is the estimated position of the central point of the vertebrae.
Based on the trained 3D characteristic point detection unit, spine center point positioning is carried out on chest CT flat scanning images of patients before operation, and in order to ensure positioning accuracy of the 3D characteristic points, positioning results are checked and corrected by a doctor with rich experience.
The data set construction process comprises the following steps:
Acquiring chest flat scanning CT image data, marking all spine center points in the image visual field, and respectively assigning numbers of 0-25 to 26 spines from C1 to S1 of a human body;
A series of intraoperative X-ray images or CT locating slice images of a corresponding patient are acquired, and long-time offline optimization calculation is performed by using a traditional registration method, so that accurate registration results are obtained, the results have high accuracy and can be used as true registration values for network training, and 3D spine center points marked in advance in CT are projected to an X-ray plane by using the registration results, so that marking of the spine center points in X-rays is completed.
The training process of the feature point detection unit is as follows:
Training a 3D feature point detection model based on the constructed CT data set marked with the spine center point; based on the established X-ray data set marked with the spine center point, a 2D characteristic point detection model is pre-trained, 2D-3D characteristic point matching is established by utilizing CT and X-rays, a pose calculation unit and a robust estimation unit with error perception are used for estimating registration results, and because all units provided by the invention can be used for micro-calculation and end-to-end training, the estimated pose is supervised by using registration true value pose, and the 2D characteristic point detection model is further trained.
The construction process of the 2D feature point detection unit is as follows:
constructing a chest X-ray data set marked with a spine central point;
And constructing and training a 2D feature point detection model based on the deep convolution neural network, wherein the basic framework is consistent with the 3D feature point detection model, and the data transmission mode of the neural network is changed from 3D to 2D.
And (3) based on the trained 2D feature point detection model, performing feature point positioning on the X-ray image acquired in real time in an operation.
The pose solving unit is constructed by the following steps:
based on the results of the 3D and 2D feature point detection units, a series of 2D-3D feature point matches are established, the gesture photographed by the C-shaped arm is utilized to provide two priori rotation angles beta and gamma, FIG. 2 is a schematic diagram of a coordinate system for registering chest CT and X-ray, two feature point matches are given, P 1 and P 2 are two 3D feature points, and the 3D feature points under the CT coordinate system are expressed as AndThe corresponding 2D feature points in the X-ray image coordinate system are represented as p 1 and p 2, and the projection equations of the 2D and 3D points in the C-arm coordinate system are as follows
Where i= {1,2}, the index representing the matching point,Representing the z-coordinate of the 3D point in the C-arm coordinate system, K represents an internal reference of the C-arm imaging system. The conversion from the CT coordinate system to the C-arm coordinate system can be expressed as
Wherein the method comprises the steps ofThe rotation and translation matrix is represented as a solving target of the registration task, which can be simplified as T. Since the rotation angle beta, gamma is known,Can be decomposed by the mode of 'xyz' compliance
In the camera coordinate system, we define two unit vectors q 1、q2, respectively representing the direction vectors from the origin to two 3D points, and the depth along the z-axis from the origin to the two 3D points is denoted μ 1、μ2, respectively, which can be obtained as follows
Wherein the method comprises the steps ofRepresenting the coordinates of a 3D point in an intermediate coordinate system, i being 1 or 2, representing the number of two 3D points, the z-axis of the intermediate coordinate system being aligned with the z-axis of the C-arm imaging coordinate system. To calculate the rotation angle alpha and translation matrixFirst an unknown depth mu 1、μ2 needs to be calculated. The calculation may be performed using the constraints provided by the two 3D points. The first constraint is that the distance between two 3D points in the intermediate coordinate system is the same as the 3D point distance in the C-arm imaging coordinate system
Since the rotation angle α is rotated only around the z-axis, the projection of the two 3D points on the z-axis of the intermediate coordinate system must be the same as in the C-arm imaging coordinate system. Thus the second constraint can be expressed as
From the above two equations, a unitary quadratic equation for μ 2 can be established so that α can be calculated according to
Wherein the method comprises the steps ofA unit vector representing a vector x is provided,The calculations can be made from mu i and q i. The translation matrix may be calculated according to the following
Preferably, the construction process of the robust estimation unit with error perception is:
based on the pose solving unit, traversing all possible feature point matching combinations, and calculating a candidate pose set
Where N is the number of vertebrae present in both CT and X-ray fields of view. After the candidate pose set is constructed, a weight is required to be allocated to each candidate pose, and the candidate pose is determined to be the correct pose or the outlier pose. For each candidate pose3D feature points in CT coordinate systemWill be transformed into the C-arm imaging coordinate system and then projected into the 2D image coordinate system according to the projection equation to obtain a series of 2D points2D feature points using 2D-SCN predictionCalculating distances between 2D feature points
For the followingThe formed distance set isTo evaluateAs confidence of correct pose, the conventional practice is to calculateThe number less than the threshold τ, however, this approach is not trivial. Thus, the present invention provides a way to calculate confidence that can be made micro. First, a valid feature point mask is calculatedFor selecting those feature points whose distance is less than a threshold value τ
Since the distance error is inversely proportional to the confidence, the confidence can be expressed as
If the denominator of the above equation is 0, then c i will be set to 0, i.e., the candidate pose will be ignored. The weight corresponding to each candidate pose can be expressed as
Normalization was performed using softmax, where λ is the temperature coefficient. For all candidate poses, its weight set is expressed asFinal robust pose estimation is achieved using weighted averaging
Wherein the method comprises the steps ofFrom the following componentsIt is deduced that since two a priori rotation angles β, γ are known, the rotation matrix can be estimated from α. Likewise, the translation matrix may be obtained by a weighted average of
Through the operation, the robust pose estimation unit with error perception is tiny, and end-to-end training can be realized.
The technical scheme of the invention is further described below with reference to the accompanying drawings and the specific embodiments.
FIG. 1 is a flow chart of a chest CT and X-ray registration algorithm based on deep learning; the invention discloses a chest CT and X-ray registration method based on deep learning, which can be used for realizing rapid initialization registration and comprises the following steps:
1. And acquiring a pre-operation chest flat scanning CT image, inputting the CT image to be registered into a 3D characteristic point detection unit, and correcting the spine central point predicted by the detection unit when a doctor performs pre-operation examination. The 3D feature point detection unit is constructed by a deep neural network, the 3D feature point positioning method is not limited, as a preferred implementation mode, a Spatial Configuration Net (SCN) is specifically used for constructing a 3D feature point detection model and training, the 3D feature point detection unit comprises a local appearance module taking 3D U-Net as a framework for realizing high-precision positioning of potential positions of all feature points, a space configuration module consisting of a multi-layer 3D convolutional neural network for realizing high-precision feature point classification, the two modules are operated in a cascade mode, and the final 3 feature point positioning result is obtained after dot product operation is carried out on output heat maps of the two modules.
2. Acquiring an intra-operative chest X-ray image, and inputting the X-ray image to be registered into a 2D feature point detection unit. The structure of the 2D feature point detection unit is basically consistent with that of the 3D feature point detection unit, and all the modules for processing the 3D image are replaced by corresponding 2D modules.
3. And 2D-3D feature point matching is established, and the 2D-3D feature point matching is input into a pose solving unit. Using the pose of the C-arm shot, two a priori rotation angles beta, gamma are provided, given two feature point matches, wherein the 3D feature point under the CT coordinate system is expressed asAndThe corresponding 2D feature points in the X-ray image coordinate system are represented as p 1 and p 2, and the projection equations of the 2D and 3D points in the C-arm coordinate system are as follows
Where i= {1,2}, the index representing the matching point,Representing the z-coordinate of the 3D point in the C-arm coordinate system, K represents an internal reference of the C-arm imaging system. The conversion from the CT coordinate system to the C-arm coordinate system can be expressed as
Wherein the method comprises the steps ofThe rotation and translation matrix is represented as a solving target of the registration task, which can be simplified as T. Since the rotation angle beta, gamma is known,Can be decomposed by the mode of 'xyz' compliance
In the camera coordinate system, we define two unit vectors q 1、q2, the depth along the z-axis from the origin to the 3D point can be noted as μ 1、μ2, and the following equation can be obtained
Wherein the method comprises the steps ofRepresenting the 3D point coordinates in an intermediate coordinate system with the z-axis aligned with the z-axis of the C-arm imaging coordinate system. To calculate the rotation angle alpha and translation matrixFirst an unknown depth mu 1、μ2 needs to be calculated. The calculation may be performed using the constraints provided by the two 3D points. The first constraint is that the distance between two 3D points in the intermediate coordinate system is the same as the 3D point distance in the C-arm imaging coordinate system
Since the rotation angle α is rotated only around the z-axis, the projection of the two 3D points on the z-axis of the intermediate coordinate system must be the same as in the C-arm imaging coordinate system. Thus the second constraint can be expressed as
From the above two equations, a unitary quadratic equation for μ 2 can be established so that α can be calculated according to
Wherein the method comprises the steps ofA unit vector representing a vector x is provided,The calculations can be made from mu i and q i. The translation matrix may be calculated according to the following
4. The set of candidate poses is input to a robust estimation unit with false perception. Based on the pose solving unit, traversing all possible feature point matching combinations, and calculating a candidate pose setWhere N is the number of vertebrae present in both CT and X-ray fields of view. After the candidate pose set is constructed, a weight is required to be allocated to each candidate pose, and the candidate pose is determined to be the correct pose or the outlier pose. For each candidate pose3D feature points in CT coordinate systemWill be transformed into the C-arm imaging coordinate system and then projected into the 2D image coordinate system according to the projection equation to obtain a series of 2D points2D feature points using 2D-SCN predictionCalculating distances between 2D feature points
For the followingThe formed distance set isTo evaluateAs confidence of correct pose, the conventional practice is to calculateThe number less than the threshold τ, however, this approach is not trivial. Thus, the present invention provides a way to calculate confidence that can be made micro. First, a valid feature point mask is calculatedFor selecting those feature points whose distance is less than a threshold value τ
Since the distance error is inversely proportional to the confidence, the confidence can be expressed as
If the denominator of the above equation is 0, then c i will be set to 0, i.e., the candidate pose will be ignored. The weight corresponding to each candidate pose can be expressed as
Normalization was performed using softmax, where λ is the temperature coefficient. For all candidate poses, its weight set is expressed asFinal robust pose estimation is achieved using weighted averaging
Wherein the method comprises the steps ofFrom the following componentsIt is deduced that since two a priori rotation angles β, γ are known, the rotation matrix can be estimated from α. Likewise, the translation matrix may be obtained by a weighted average of
Through the operation, the robust pose estimation unit with error perception is tiny, and end-to-end training can be realized.
In this embodiment, the training steps are as follows:
Constructing a marked CT characteristic point detection data set and training a 3D characteristic point detection unit. In the embodiment, a 3D convolutional neural network is trained by using a public CT characteristic point detection dataset VerSe;
Constructing a marked X-ray characteristic point detection data set and training a 2D characteristic point detection unit. A series of CT images of the same patient and corresponding X-ray films are acquired, a trained 3D characteristic point detection unit is utilized to detect a spine central point in the CT, a traditional registration algorithm is utilized to optimize the CT and the X-ray images for a long time, an accurate registration result is obtained, the result is used as a registration truth value, and the characteristic points detected in the CT are projected onto an X-ray plane by utilizing the registration result, so that labeling of the spine central point in the X-ray is obtained. Based on these 2D labels, a 2D convolutional neural network is trained using root mean square loss function L MSE, resulting in a pre-trained 2D texel detection unit. The method comprises the steps of utilizing CT and X-ray to establish 2D-3D characteristic point matching, estimating a registration result by using a pose calculation unit and a robust estimation unit with error perception, and calculating a 1-norm loss function L pose by using the pose estimation result and the registration truth value.
In an embodiment, the proposed algorithm is built by using a deep learning framework PyTorch, training is performed using Adam optimizer, learning rate is 0.0001, epsilon is set to 0.001, X-rays are uniformly cut into images with resolution 256×256 according to the same physical pixel size, and X-rays and CT datasets are cut according to 3:1:1, a training set, a validation set and a test set.
In order to prove the superiority of the positioning algorithm, a registration experiment is designed by using a real data set to compare with a plurality of registration methods commonly used at present, and the experimental result can be seen in table 1, which is the comparison experimental result of the positioning algorithm and other registration methods.
The first column of the table represents an abbreviation for the different methods. Wherein, opt-NGI is a method based on optimization and normalization of gradient information similarity proposed by A Uneri et al in article 3D-2D registration for surgical guidance:effect of projection view angles on registration accuracy, opt-GO and Opt-GC are optimization methods based on gradient direction and gradient cross correlation measurement proposed by TDe Silva et al in article 3D–2D image registration for target localization in spine surgery:investigation of similarity metrics providing robustness to content mismatch. SVR is a global sample based deep learning method proposed by Benjamin Hou et al in article PREDICTING SLICE-to-Volume Transformation IN PRESENCE of Arbitrary Subject Motion. 2P-EARE is the method of the invention. The second column to the last column show a rotation angle error, a translation error in three directions, a comprehensive target registration error, and a registration failure rate of the registration method, respectively. The criterion for failure is that the target registration error is greater than 30mm. From the experimental results, the registration method can obtain higher comprehensive registration accuracy and higher success rate of registration.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (6)
1. The chest CT and X-ray real-time registration method based on deep learning is characterized by comprising the following steps of:
1) Acquiring a three-dimensional chest CT flat scanning image of a patient before an operation, positioning a spine center point in CT by using a 3D feature point detection unit, checking a positioning result of the 3D feature point, and ensuring the precision of the 3D feature point;
2) Acquiring a two-dimensional chest X-ray image of a patient in an operation, positioning a spine center point in X-rays by using a 2D feature point detection unit, and establishing a series of 2D-3D feature point correlations by using 3D feature points positioned in advance;
3) Inputting the established association points and the shooting gestures of the C-shaped arm into a gesture solving unit, and solving a 6-degree-of-freedom candidate gesture by using association of every two feature points;
4) Inputting all the solved candidate poses into a robust estimation unit with error perception, estimating feature points positioned in error at the same time, and aggregating all the candidate poses to obtain robust and accurate poses;
The robust estimation unit with error perception comprises the following construction processes:
based on the pose solving unit, traversing all possible feature point matching combinations, and calculating a candidate pose set Wherein N is the number of spines appearing in CT and X-ray fields at the same time, after the construction of the candidate pose set is completed, weight is allocated to each candidate pose, and the candidate pose is determined to be the correct pose or the outlier pose, and for each candidate pose3D feature points in CT coordinate systemWill be transformed into the C-arm imaging coordinate system and then projected into the 2D image coordinate system according to the projection equation to obtain a series of 2D points2D feature points using 2D-SCN predictionCalculating the distance between 2D feature points:
for the following The formed distance set isEvaluation by way of a plausible calculation of the confidence levelAs confidence of correct pose, a valid feature point mask is calculatedFor selecting those feature points whose distance is less than a threshold value τ
Since the distance error is inversely proportional to the confidence, the confidence can be expressed as
If the denominator is 0, then c i will be set to 0, i.e. the candidate pose will be ignored, and the weight corresponding to each candidate pose can be expressed as
Normalization was performed using softmax, where λ is the temperature coefficient, and for all candidate poses, the set of weights is expressed asFinal robust pose estimation is achieved using weighted averaging
Wherein the method comprises the steps ofFrom the following componentsIt is deduced that since two a priori rotation angles β, γ are known, the rotation matrix can be estimated from α, and likewise the translation matrix can be obtained by weighted averaging as follows
Through the operation, the robust pose estimation unit with error perception is tiny, and end-to-end training is realized.
2. The chest CT and X-ray real-time registration method based on deep learning as set forth in claim 1, wherein the 3D feature point detection unit includes the following construction process:
constructing a chest flat scanning CT data set marked with a spine central point;
Constructing a 3D feature point detection model based on a deep convolutional neural network, training by a network training method, outputting a thermodynamic diagram with N channels, the size of which is the same as that of an original image, by the network, wherein channel serial numbers represent the types of feature points, namely N vertebrae forming a human spine, and the position with the maximum thermodynamic diagram response in each channel is the estimated position of a spine center point;
Based on the trained 3D characteristic point detection model, spine center point positioning is carried out on chest CT flat scanning images of patients before operation, and in order to ensure positioning accuracy of the 3D characteristic points, positioning results are checked and corrected.
3. The chest CT and X-ray real-time registration method based on deep learning as claimed in claim 2, wherein the data set construction method comprises the steps of:
Acquiring chest flat scanning CT image data, marking all spine center points in an image view, and respectively assigning numbers of 0-N to N vertebrae of a human body;
collecting a series of intraoperative X-ray images or CT locating slice images of a corresponding patient, and performing long-time offline optimization calculation by using a traditional registration method, so as to obtain an accurate registration result, wherein the accurate registration result is used as a true value of registration for network training;
and (3) projecting the 3D spine central point marked in advance in CT to an X-ray plane by using the registration results, thereby finishing marking the spine central point in X-ray.
4. A deep learning based chest CT and X-ray real-time registration method according to claim 2 or 3, wherein said network training method comprises the steps of:
training a 3D feature point detection model based on the constructed CT data set marked with the spine center point;
Based on the established X-ray data set marked with the spine center point, a 2D characteristic point detection model is pre-trained, 2D-3D characteristic point matching is established by utilizing CT and X-rays, registration results are estimated by using a pose calculation unit and a robust estimation unit with error perception, end-to-end training is carried out, the estimated poses are supervised by using registration truth poses, and the 2D characteristic point detection model is further trained.
5. A chest CT and X-ray real-time registration method based on deep learning as claimed in claim 1, 2 or 3, wherein said 2D feature point detection unit comprises the following construction process:
constructing a chest X-ray data set marked with a spine central point;
2D feature point detection models are built and trained based on the deep convolutional neural network, the basic framework of the 2D feature point detection models is consistent with the 3D feature point detection models, and the data transmission mode of the neural network is changed from 3D to 2D;
And (3) based on the trained 2D feature point detection model, performing feature point positioning on the X-ray image acquired in real time in an operation.
6. The chest CT and X-ray real-time registration method based on deep learning as set forth in claim 5, wherein the pose solving unit includes the following construction process:
Based on the results of the 3D and 2D feature point detection units, a series of 2D-3D feature point matches are established, the gesture photographed by the C-shaped arm is utilized to provide two priori rotation angles beta and gamma, and two feature point matches are given, wherein the 3D feature points under the CT coordinate system are expressed as AndThe corresponding 2D feature points in the X-ray image coordinate system are represented as p 1 and p 2, and the projection equations of the 2D and 3D points in the C-arm coordinate system are as follows
Where i= {1,2}, the index representing the matching point,Representing the z-coordinate of the 3D point in the C-arm coordinate system, K represents the internal reference of the C-arm imaging system, and the conversion relation from the CT coordinate system to the C-arm coordinate system is expressed as
Wherein the method comprises the steps ofRepresenting the rotation and translation matrices, which are the solving targets for the registration task, is simplified as T, since the rotation angles beta, gamma are known,Can be decomposed by the mode of 'xyz' compliance
In the camera coordinate system, two unit vectors q 1、q2 are defined, which respectively represent direction vectors from the origin to two 3D points, and depths along the z-axis from the origin to the two 3D points are respectively denoted as μ 1、μ2, resulting in the following equation
Wherein the method comprises the steps ofRepresenting the coordinates of a 3D point in an intermediate coordinate system, i being 1 or 2, representing the numbering of two 3D points, the z-axis of the intermediate coordinate system being aligned with the z-axis of the C-arm imaging coordinate system, in order to calculate the rotation angle α and the translation matrixFirstly, an unknown depth mu 1、μ2 needs to be calculated, and the calculation is carried out by using the constraint provided by two 3D points, wherein the first constraint is that the distance between the two 3D points in an intermediate coordinate system is the same as the distance between the 3D points in a C-shaped arm imaging coordinate system
Since the rotation angle α is rotated only around the z-axis, the projection of the two 3D points on the z-axis of the intermediate coordinate system must be the same as in the C-arm imaging coordinate system, the second constraint being expressed as
Based on the above constraint, a unitary quadratic equation for μ 2 is established so that α is calculated according to
Wherein the method comprises the steps ofA unit vector representing a vector x is provided,Calculated according to mu i and q i, the translation matrix is calculated according to the following formula
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111167612.6A CN113870331B (en) | 2021-10-07 | 2021-10-07 | Chest CT and X-ray real-time registration algorithm based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111167612.6A CN113870331B (en) | 2021-10-07 | 2021-10-07 | Chest CT and X-ray real-time registration algorithm based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113870331A CN113870331A (en) | 2021-12-31 |
| CN113870331B true CN113870331B (en) | 2024-07-26 |
Family
ID=79001823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111167612.6A Active CN113870331B (en) | 2021-10-07 | 2021-10-07 | Chest CT and X-ray real-time registration algorithm based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113870331B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115205417B (en) * | 2022-09-14 | 2023-03-14 | 首都医科大学附属北京安贞医院 | Projection transformation calculation method, device, equipment and storage medium |
| CN116958075A (en) * | 2023-07-10 | 2023-10-27 | 宁波大学 | Spine key point detection system |
| CN117017487B (en) * | 2023-10-09 | 2024-01-05 | 杭州键嘉医疗科技股份有限公司 | Spinal column registration method, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106127753A (en) * | 2016-06-20 | 2016-11-16 | 中国科学院深圳先进技术研究院 | CT image body surface handmarking's extraction method in a kind of surgical operation |
| CN112184782A (en) * | 2020-09-21 | 2021-01-05 | 上海交通大学 | Automatic registration positioning method and device for bone joint |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2932470A2 (en) * | 2013-10-18 | 2015-10-21 | Koninklijke Philips N.V. | Registration of medical images |
-
2021
- 2021-10-07 CN CN202111167612.6A patent/CN113870331B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106127753A (en) * | 2016-06-20 | 2016-11-16 | 中国科学院深圳先进技术研究院 | CT image body surface handmarking's extraction method in a kind of surgical operation |
| CN112184782A (en) * | 2020-09-21 | 2021-01-05 | 上海交通大学 | Automatic registration positioning method and device for bone joint |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113870331A (en) | 2021-12-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113870331B (en) | Chest CT and X-ray real-time registration algorithm based on deep learning | |
| JP5797352B1 (en) | Method for tracking a three-dimensional object | |
| US10318839B2 (en) | Method for automatic detection of anatomical landmarks in volumetric data | |
| US10507002B2 (en) | X-ray system and method for standing subject | |
| US8271068B2 (en) | Method for dynamic road mapping | |
| Luó et al. | Development and comparison of new hybrid motion tracking for bronchoscopic navigation | |
| TWI785588B (en) | Image registration method and related model training methods, equipment and computer readable storage medium thereof | |
| KR101767005B1 (en) | Method and apparatus for matching images using contour-based registration | |
| Varnavas et al. | Increasing the automation of a 2D-3D registration system | |
| US10078906B2 (en) | Device and method for image registration, and non-transitory recording medium | |
| CN113112486B (en) | Tumor motion estimation method and device, terminal equipment and storage medium | |
| Su et al. | Comparison of 3d surgical tool segmentation procedures with robot kinematics prior | |
| Park et al. | Accurate and efficient 3d human pose estimation algorithm using single depth images for pose analysis in golf | |
| Shao et al. | Augmented reality calibration using feature triangulation iteration-based registration for surgical navigation | |
| Kang et al. | Robustness and accuracy of feature-based single image 2-D–3-D registration without correspondences for image-guided intervention | |
| Meng et al. | 2D-3D registration with weighted local mutual information in vascular interventions | |
| Zampokas et al. | Real-time 3D reconstruction in minimally invasive surgery with quasi-dense matching | |
| CN111080676A (en) | Method for tracking endoscope image sequence feature points through online classification | |
| CN117159138B (en) | Joint positioning-based 2D-3D registration method and system | |
| Zheng et al. | Surface reconstruction of bone from X-ray images and point distribution model incorporating a novel method for 2D-3D correspondence | |
| JP7498404B2 (en) | Apparatus, method and program for estimating three-dimensional posture of subject | |
| CN108804861B (en) | Spinal minimally invasive surgery training system and method with true force feedback | |
| CN112053330A (en) | Diaphragm prediction system and method based on PCA (principal component analysis) and TSSM (time series short message System) models | |
| CN112330603A (en) | System and method for estimating motion of target in tissue based on soft tissue surface deformation | |
| Yang et al. | A novel neurosurgery registration pipeline based on heat maps and anatomic facial feature points |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |