CN115861662A - Prediction method, device, equipment and medium based on combined neural network model - Google Patents
Prediction method, device, equipment and medium based on combined neural network model Download PDFInfo
- Publication number
- CN115861662A CN115861662A CN202310148171.8A CN202310148171A CN115861662A CN 115861662 A CN115861662 A CN 115861662A CN 202310148171 A CN202310148171 A CN 202310148171A CN 115861662 A CN115861662 A CN 115861662A
- Authority
- CN
- China
- Prior art keywords
- neural network
- sub
- network model
- image data
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003062 neural network model Methods 0.000 title claims abstract description 203
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000012549 training Methods 0.000 claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 18
- 238000012795 verification Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 230000004927 fusion Effects 0.000 claims description 5
- 238000010200 validation analysis Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000002790 cross-validation Methods 0.000 description 6
- 230000006854 communication Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 238000002591 computed tomography Methods 0.000 description 2
- 238000010968 computed tomography angiography Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 206010061216 Infarction Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000005013 brain tissue Anatomy 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000007574 infarction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000010412 perfusion Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a prediction method, a device, equipment and a medium based on a combined neural network model, wherein the combined neural network model comprises a plurality of different sub-neural network models, and the method comprises the following steps: acquiring image data to be processed; processing the image data to be processed by utilizing each sub neural network model respectively to obtain corresponding prediction results; respectively calculating the similarity between the image data to be processed and the template image corresponding to each sub-neural network model, wherein the template image is determined based on a training image data set of the sub-neural network model corresponding to training; and based on the similarity between the image data to be processed and each template image, fusing the prediction results corresponding to each sub-neural network model by adopting a weighted average method to obtain a final prediction result. The prediction method improves the prediction accuracy of the combined neural network model by dynamically distributing the weight coefficient.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a prediction method, a prediction device, prediction equipment and a prediction medium based on a combined neural network model.
Background
Deep learning is a branch of machine learning, and aims to establish a neural network simulating human brain for analysis and learning and process data by simulating the working mechanism of the human brain. Deep learning is widely used in the field of medical image processing, and has made great progress in almost all applications (e.g., segmentation, detection, classification, registration, etc.).
In practical application, in order to obtain a more stable and reliable prediction result, a plurality of neural network models may be trained based on a training image data set, and then outputs of the respective neural network models are combined by using a model combination strategy to obtain a final prediction result. Taking the most common model combination strategy at present, k-fold cross validation as an example, the basic steps are as follows: 1) Randomly dividing a training image data set into k groups; 2) Taking one group as a verification data set by adopting a leave-one method, and taking the remaining k-1 groups as a training data set to carry out deep learning training to obtain k neural network models; 3) And averaging the outputs of the k neural network models to obtain a final prediction result.
However, the model combination strategy for evenly distributing the weight coefficients ignores the difference of generalization ability among the neural network models, so that the final prediction result is sacrificed in terms of accuracy, and a space for further improvement exists.
Disclosure of Invention
In view of the above problems in the prior art, an object of the present invention is to provide a prediction method, apparatus, device and medium based on a combined neural network model, which can improve the prediction accuracy of the combined neural network model.
In order to solve the above problem, the present invention provides a prediction method based on a combined neural network model, the combined neural network model including a plurality of different sub-neural network models, the method including:
acquiring image data to be processed;
processing the image data to be processed by utilizing each sub neural network model respectively to obtain corresponding prediction results;
respectively calculating the similarity between the image data to be processed and the template image corresponding to each sub neural network model, wherein the template image is determined based on a training image data set of the sub neural network model corresponding to training;
and based on the similarity between the image data to be processed and each template image, fusing the prediction results corresponding to each sub-neural network model by adopting a weighted average method to obtain a final prediction result.
Further, the respectively calculating the similarity between the image data to be processed and the template image corresponding to each sub neural network model includes:
respectively carrying out non-rigid registration on the image data to be processed and the template image aiming at each template image to obtain a loss value after the registration is finished;
and determining the similarity between the image data to be processed and the template image according to the loss value.
Further, the calculation formula of the similarity between the image data to be processed and the template image is as follows:
wherein S is i Similarity between the image data to be processed and a template image corresponding to the ith sub-neural network model, L i And registering the image data to be processed and the template image corresponding to the ith sub-neural network model to obtain a loss value after the registration is completed.
Further, the fusing the prediction results corresponding to the respective sub-neural network models by using a weighted average method based on the similarity between the image data to be processed and the respective template images to obtain a final prediction result includes:
based on a preset weight model, respectively calculating weight coefficients corresponding to prediction results corresponding to the sub-neural network models according to the similarity between the image data to be processed and the template images;
and carrying out weighted average on the prediction results corresponding to the sub-neural network models according to the weight coefficients to obtain the final prediction result.
Further, a calculation formula of the weight coefficient corresponding to the prediction result corresponding to each sub-neural network model is as follows:
wherein, ω is i Weight coefficient corresponding to prediction result corresponding to ith sub-neural network model, k is number of sub-neural network models, S i For similarity between the image data to be processed and the template image corresponding to the ith sub-neural network model, NS i Is normalized similarity.
Further, the method further comprises:
determining a training image data set for training each sub-neural network model respectively;
and constructing a template image corresponding to the sub-neural network model according to the training image data set.
Further, the training image dataset comprises a training dataset and a validation dataset;
the constructing of the template image corresponding to the sub-neural network model according to the training image dataset includes:
constructing a template image corresponding to the sub-neural network model according to the training data set;
or,
and constructing a template image corresponding to the sub-neural network model according to the verification data set.
Another aspect of the present invention provides a prediction apparatus based on a combined neural network model, the combined neural network model including a plurality of different sub-neural network models, the apparatus including:
the acquisition module is used for acquiring image data to be processed;
the processing module is used for processing the image data to be processed by utilizing each sub-neural network model respectively to obtain a corresponding prediction result;
the calculation module is used for calculating the similarity between the image data to be processed and the template images corresponding to the sub neural network models respectively, and the template images are determined based on training image data sets of the sub neural network models corresponding to training;
and the fusion module is used for fusing the prediction results corresponding to the sub-neural network models by adopting a weighted average method based on the similarity between the image data to be processed and each template image to obtain the final prediction result.
Another aspect of the present invention provides an electronic device, including a processor and a memory, where at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded by the processor and executed to implement the method as described above.
Another aspect of the present invention provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the method as described above.
Due to the technical scheme, the invention has the following beneficial effects:
according to the prediction method provided by the embodiment of the invention, the accuracy of each sub-neural network model is pre-evaluated by calculating the similarity between the image data to be processed and the template image corresponding to each sub-neural network model in the combined neural network model, and the weight coefficient corresponding to the prediction result corresponding to each sub-neural network model is dynamically distributed based on the evaluation result, so that the prediction accuracy of the combined neural network model can be improved, and the method can be used in various neural network models using model combination strategies and has wide applicability.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings used in the description of the embodiment or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a flow chart of a prediction method based on a combined neural network model according to an embodiment of the present invention;
FIG. 2 is a flow chart of a prediction method based on a combined neural network model according to another embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a prediction apparatus based on a combined neural network model according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a prediction apparatus based on a combined neural network model according to another embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or device.
Referring to the specification, fig. 1 shows a flow of a prediction method based on a combined neural network model, which is a model using a model combination strategy and may include a plurality of different sub neural network models, according to an embodiment of the present invention. The method can be applied to a server, the server can be an independently operated server, or a server cluster or distributed system consisting of a plurality of servers, and can also be a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content Delivery Network (CDN), big data, an artificial intelligence platform and the like.
As shown in fig. 1, the method may include the following steps:
s110: and acquiring image data to be processed.
The method provided by the embodiment of the invention can be applied to scenes for image processing based on the combined neural network model, such as scenes for image recognition, image segmentation and the like based on the combined neural network model. In different scenes, the image data to be processed can be different types of image data, and the combined neural network model can also be different combined neural network models. The different combined neural network models refer to combined neural network models comprising at least one different sub-neural network model, and the different sub-neural network models refer to models with different network structures or different network parameters.
Illustratively, the image data to be processed may be acquired medical image data of a target tissue of the patient, such as Non-Contrast computed Tomography (NCCT) image data, computed Tomography Perfusion (CTP) image data, or Computed Tomography Angiography (CTA) image data of the brain tissue. The combined neural network model may include a model based on a plurality of different sub-neural network models trained by a Unet network, a Unet + + network, a Vnet network, a dense-Vnet network, and/or a transform-Unet network. The combined neural network model can process the medical image data to obtain a predicted low-density infarct area prediction result.
S120: and processing the image data to be processed by utilizing each sub-neural network model respectively to obtain corresponding prediction results.
In the embodiment of the invention, because the combined neural network model comprises a plurality of different sub-neural network models, the image data to be processed can be respectively input into each sub-neural network model for processing, each sub-neural network model can be processed to obtain a corresponding prediction result, and then the prediction results output by each sub-neural network model are fused by adopting a model combination strategy, so that the final prediction result can be obtained.
Exemplarily, assuming that the combined neural network model includes k different sub-neural network models, the k different sub-neural network models may be used to process the image data to be processed one by one to obtain a corresponding prediction result P i ,i=1,2,...,k。
In the prior art, the prediction results output by the respective sub-neural network models are generally averaged (that is, the same weighting coefficient 1/k is given to each sub-neural network model, and k is the number of the sub-neural network models) to obtain a final prediction result. However, because the generalization ability of each sub-neural network model has a certain difference, it is difficult to ensure the accuracy of the finally obtained prediction result in an average manner, resulting in poor generalization ability and reliability of the overall model.
On the aspect of defining a data set, improving the network structure of the model can improve the accuracy of a prediction result, but the generalization capability of the model on new data is improved, and more is the richness problem of the data of the training set, but not the network structure problem. Therefore, the richness of the training data set is ensured, the training data set contains various boundary conditions as much as possible, and the generalization capability of almost all models can be greatly improved. In other words, the reliability of the model can be inferred by quantifying how similar the new data is to the training data set. Based on this, the embodiment of the present invention may dynamically allocate the weight of each sub neural network model by calculating the image similarity driven by the data set, so as to improve the accuracy of the prediction result of the model combination strategy, and the specific implementation flow is as follows steps S130 and S140.
In a possible embodiment, before performing the step S120, a step of training the combined neural network model may be further included. Specifically, a plurality of preset neural networks can be subjected to deep learning training respectively through a training image data set obtained in advance to obtain a plurality of sub-neural network models, so that the combined neural network model can be obtained through combination. The preset neural network may include, but is not limited to, a pnet network, a pnet + + network, a Vnet network, a dense-Vnet network, a transform-pnet network, and the like, which is not limited in this embodiment of the present invention.
Illustratively, a pre-acquired training image data set can be divided into k groups in a k-fold cross validation mode, any k-1 group is taken as a training data set each time, the rest group is a test data set, a preset neural network is trained by using the training image data in the training data set, the performance of a trained sub-neural network model is tested by using the training image data in the test data set, the performance of the trained sub-neural network model is evaluated, and finally a reliable sub-neural network model can be obtained. K different sub-neural network models can be obtained through training in a k-fold cross validation mode, and the k sub-neural network models can be combined to obtain the combined neural network model. The value of k may be preset according to actual needs, for example, k may be generally set to 3,5, or 10 in consideration of the balance between the prediction speed and the accuracy of the model combination strategy, which is not limited in this embodiment of the present invention.
It should be noted that the above embodiment of performing model training by using a k-fold cross validation method is only an example, and in practical applications, a plurality of different sub-neural network models may also be obtained by using other methods for training, which is not limited in this embodiment of the present invention. For example, under the condition that the data volume of the training image data is sufficient, a pre-acquired training image data set can be divided into k groups, each group of training image data is divided into a training data set and a test data set, a sub-neural network model is obtained through training, in this way, k sub-neural network models can be obtained through training, and finally the combined neural network model is obtained through combination.
S130: and respectively calculating the similarity between the image data to be processed and the template image corresponding to each sub-neural network model, wherein the template image is determined based on a training image data set for training the corresponding sub-neural network model.
In a possible embodiment, before performing step S130, a step of constructing a template image corresponding to each of the sub neural network models may further be included, and the step of constructing a template image corresponding to each of the sub neural network models may include:
determining a training image data set for training each sub-neural network model respectively;
and constructing a template image corresponding to the sub-neural network model according to the training image data set.
Since the training image data set includes a training data set and a verification data set, it is finally required to determine the similarity between the image data to be processed and the training data set to estimate the reliability of the model. Therefore, in an alternative embodiment, the template image may be obtained by fusing training data sets corresponding to the respective sub neural network models to identify the training data sets. Specifically, the constructing of the template image corresponding to the sub neural network model according to the training image data set may include: and constructing a template image corresponding to the sub-neural network model according to the training data set.
It can be understood that the template image corresponding to each sub-neural network model is obtained by utilizing the fusion of the training data sets, and the training data sets of each sub-neural network model can be quantized into one template image, so that the similarity can be calculated subsequently, and the prediction efficiency of the combined neural network model can be improved.
Under the condition of model training in a k-fold cross validation mode, k-2 groups of repeated training image data exist in training data sets corresponding to every two sub neural network models, so that two template images with high similarity can be constructed, and the accuracy of subsequent calculation results is influenced. Therefore, in another alternative embodiment, the template image may be obtained by fusing the verification data sets corresponding to the respective sub neural network models to identify the verification data sets. Specifically, the constructing the template image corresponding to the sub neural network model according to the training image dataset may include: and constructing a template image corresponding to the sub-neural network model according to the verification data set.
In practical application, an open source tool such as The Advanced Normalization Tools and The like can be used to generate a corresponding template image based on The training data set or The verification data set corresponding to each sub-neural network model.
It can be understood that, under the condition of model training by adopting a k-fold cross validation mode, the template images corresponding to the sub-neural network models are obtained by utilizing the fusion of validation data sets, so that the difference of the template images corresponding to the sub-neural network models can be increased, the accuracy of the pre-evaluation result of each sub-neural network model is improved, and the prediction effect of the combined neural network model is improved.
In the embodiment of the invention, the template image (which can be recorded as T) corresponding to each sub-neural network model is obtained i I =1, 2.. K), the image data to be processed and the template images T can be calculated separately i The similarity between them. In view of the image to be processed T i The displacement and deformation of the data and each template image possibly exist can convert the problem of calculating the similarity into the registration problem.
Specifically, the respectively calculating the similarity between the image data to be processed and the template image corresponding to each of the sub neural network models may include:
respectively carrying out non-rigid registration on the image data to be processed and the template images aiming at each template image to obtain a loss value after the registration is finished;
and determining the similarity between the image data to be processed and the template image according to the loss value.
In the embodiment of the present invention, the image data to be processed and each template image T may be registered by various non-rigid registration methods in the prior art i The non-rigid registration is performed, and the embodiment of the present invention is not described herein again.
In the embodiment of the present invention, various methods in the prior art, for example, calculation methods such as mean square error, normalized correlation coefficient, mutual information, and normalized mutual information, may be adopted to calculate the loss value (which may be denoted as L) i I =1,2, k), embodiments of the invention are not described in detail here.
In the embodiment of the present invention, the similarity between the image data to be processed and the template image may be determined according to the calculated loss value. Specifically, the smaller the calculated loss value is, the greater the similarity between the image data to be processed and the template image is.
In one possible embodiment, the similarity between the image data to be processed and the template image may be determined by the following calculation formula:
wherein S is i Similarity between image data to be processed and template images corresponding to the ith sub-neural network model, L i And registering the image data to be processed and the template image corresponding to the ith sub-neural network model to obtain a loss value after the registration is completed.
It should be noted that, the embodiment of calculating the similarity between the image data to be processed and each of the template images according to the formula (1) is merely an example, and in practical applications, the similarity between the image data to be processed and each of the template images may also be determined in other manners, for example, the similarity may be determined according to a predetermined correspondence between a loss value and a similarity and a calculated loss value, which is not limited in this embodiment of the present invention.
It can be understood that the similarity problem between the image data to be processed and the template images corresponding to the respective sub neural network models is converted into a registration problem, so that the accuracy of the similarity calculation result can be improved, and the accuracy of the finally obtained prediction result can be further improved.
S140: and based on the similarity between the image data to be processed and each template image, fusing the prediction results corresponding to each sub-neural network model by adopting a weighted average method to obtain a final prediction result.
In the embodiment of the invention, the reliability of each sub-neural network model can be inferred by quantifying the similarity degree of the image data to be processed and the training data set corresponding to each sub-neural network model, and the weight of the prediction result corresponding to each sub-neural network model is further dynamically distributed, so that the accuracy of the final prediction result is improved.
In a possible embodiment, with reference to fig. 2 of the specification, the performing, based on the similarity between the image data to be processed and each of the template images, fusion processing on the prediction results corresponding to each of the sub-neural network models by using a weighted average method to obtain a final prediction result may include:
s141: and respectively calculating the weight coefficients corresponding to the prediction results corresponding to the sub-neural network models according to the similarity between the image data to be processed and the template images based on a preset weight model.
Specifically, when the template image corresponding to each sub-neural network model is constructed based on a verification data set, the greater the similarity between the image data to be processed and the template image, the closer the verification data set of the sub-neural network model corresponding to the image data to be processed and the template image is, the closer the verification data set is to the training data set, and therefore the weighting coefficient corresponding to the prediction result corresponding to the sub-neural network model should be smaller.
For example, the weight coefficient corresponding to the prediction result corresponding to each of the sub neural network models may be obtained by the following calculation formula:
wherein, ω is i Weight coefficient corresponding to prediction result corresponding to ith sub-neural network model, k is number of sub-neural network models, S i For similarity between the image data to be processed and the template image corresponding to the ith sub-neural network model, NS i Is the normalized similarity.
Specifically, in a case where the template image corresponding to each of the sub neural network models is constructed based on a training data set, the greater the similarity between the image data to be processed and the template image, the closer the training data set of the sub neural network model corresponding to the image data to be processed and the template image is, the greater the weight coefficient corresponding to the prediction result corresponding to the sub neural network model should be.
It should be noted that the weight model may be preset according to an actual situation, and only needs to make the similarity and the weight coefficient have an inverse correlation relationship when the template image corresponding to each sub neural network model is constructed based on the verification data set, and make the similarity and the weight coefficient have a positive correlation relationship when the template image corresponding to each sub neural network model is constructed based on the training data set, which is not limited in this embodiment of the present invention.
It can be understood that the accuracy of each sub-neural network model is pre-evaluated by utilizing the similarity between the template image and the image data to be processed, and a higher weight coefficient is given to the sub-neural network models which are well performed, so that the prediction accuracy of the combined neural network model and the generalization capability of the model can be improved.
S142: and carrying out weighted average on the prediction results corresponding to the sub-neural network models according to the weight coefficients to obtain the final prediction result.
In the embodiment of the present invention, after the weight coefficient of the prediction result corresponding to each sub-neural network model is obtained through calculation, a weighted average of the prediction results corresponding to each sub-neural network model may be taken as a final prediction result, and a specific calculation formula is as follows:
where P is the final prediction, ω j Weight coefficient corresponding to prediction result corresponding to jth sub-neural network model, k is number of sub-neural network models, P j And the prediction result corresponding to the jth sub-neural network model.
In summary, according to the prediction method of the embodiment of the present invention, the accuracy of each sub-neural network model is pre-evaluated by calculating the similarity between the image data to be processed and the template image corresponding to each sub-neural network model in the combined neural network model, and the weight coefficient corresponding to the prediction result corresponding to each sub-neural network model is dynamically allocated based on the evaluation result, so that the prediction accuracy of the combined neural network model can be improved, and the method can be used in various neural network models using model combination strategies, and has wide applicability.
Referring to the specification, fig. 3 illustrates a structure of a prediction apparatus 300 based on a combined neural network model according to an embodiment of the present invention, where the combined neural network model may include a plurality of different sub-neural network models, and as shown in fig. 3, the apparatus 300 may include:
an obtaining module 310, configured to obtain image data to be processed;
the processing module 320 is configured to process the image data to be processed by using each sub neural network model, so as to obtain a corresponding prediction result;
a calculating module 330, configured to calculate similarities between the image data to be processed and template images corresponding to the respective sub neural network models, where the template images are determined based on training image data sets of the sub neural network models corresponding to training;
and the fusion module 340 is configured to perform fusion processing on the prediction results corresponding to each sub neural network model by using a weighted average method based on the similarity between the image data to be processed and each template image, so as to obtain a final prediction result.
In one possible embodiment, as shown in fig. 4, the apparatus 300 may further include:
a determining module 350, configured to determine, for each of the sub-neural network models, a training image data set for training the sub-neural network model;
the building module 360 is configured to build a template image corresponding to the sub neural network model according to the training image data set.
In one possible embodiment, the apparatus 300 may further include:
and the model training module is used for training the combined neural network model.
It should be noted that, when the apparatus provided in the foregoing embodiment implements the functions thereof, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to implement all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and specific implementation processes thereof are detailed in the corresponding method embodiments and are not described herein again.
An embodiment of the present invention further provides an electronic device, which includes a processor and a memory, where at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the prediction method based on the combined neural network model provided in the above method embodiment.
The memory may be used to store software programs and modules, and the processor may execute various functional applications and data processing by operating the software programs and modules stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs needed by functions and the like; the storage data area may store data created according to use of the apparatus, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the processor access to the memory.
In a specific embodiment, fig. 5 is a schematic diagram illustrating a hardware structure of an electronic device for implementing the prediction method based on the combined neural network model according to the embodiment of the present invention, where the electronic device may be a computer terminal, a mobile terminal, or another device, and the electronic device may also participate in forming or including the prediction apparatus based on the combined neural network model according to the embodiment of the present invention. As shown in fig. 5, the electronic device 500 may include one or more computer-readable storage media including a memory 510, one or more processing cores including a processor 520, an input unit 530, a display unit 540, a Radio Frequency (RF) circuit 550, a wireless fidelity (WiFi) module 560, and a power supply 570. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 5 does not constitute a limitation of electronic device 500, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components. Wherein:
the memory 510 may be used to store software programs and modules, and the processor 520 may execute various functional applications and data processing by operating or executing the software programs and modules stored in the memory 510 and calling data stored in the memory 510. The memory 510 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory 510 may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device. Accordingly, memory 510 may also include a memory controller to provide processor 520 with access to memory 510.
The processor 520 is a control center of the electronic device 500, connects various parts of the whole electronic device using various interfaces and lines, and performs various functions of the electronic device 500 and processes data by operating or executing software programs and/or modules stored in the memory 510 and calling data stored in the memory 510, thereby performing overall monitoring of the electronic device 500. The Processor 520 may be a central processing unit, or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input unit 530 may be used to receive input numeric or character information and generate a keyboard, mouse, joystick, optical or trackball signal input related to user setting and function control. In particular, the input unit 530 may include a touch sensitive surface 531 as well as other input devices 532. In particular, the touch-sensitive surface 531 may include, but is not limited to, a touch pad or touch screen, and the other input devices 532 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The display unit 540 may be used to display information input by or provided to a user and various graphical user interfaces of the electronic device, which may be made up of graphics, text, icons, video, and any combination thereof. The Display unit 540 may include a Display panel 541, and optionally, the Display panel 541 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.
The RF circuit 550 may be used for receiving and transmitting signals during a message transmission or communication process, and in particular, for receiving downlink information of a base station and then processing the received downlink information by the one or more processors 520; in addition, data relating to uplink is transmitted to the base station. In general, RF circuitry 550 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, RF circuit 550 may also communicate with networks and other devices via wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), general Packet Radio Service (GPRS), code Division Multiple Access (CDMA), wideband Code Division Multiple Access (WCDMA), long Term Evolution (LTE), email, short Messaging Service (SMS), and the like.
WiFi belongs to short-range wireless transmission technology, and the electronic device 500 can help the user send and receive e-mails, browse web pages, access streaming media, etc. through the WiFi module 560, which provides the user with wireless broadband internet access. Although fig. 5 shows the WiFi module 560, it is understood that it does not belong to the essential constitution of the electronic device 500, and may be omitted entirely as needed within the scope not changing the essence of the invention.
The electronic device 500 further includes a power supply 570 (e.g., a battery) for supplying power to the various components, and preferably, the power supply is logically connected to the processor 520 via a power management system, so that functions of managing charging, discharging, and power consumption are implemented via the power management system. Power supply 570 may also include any component including one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.
It should be noted that, although not shown, the electronic device 500 may further include a bluetooth module, and the like, which are not described herein again.
An embodiment of the present invention further provides a computer-readable storage medium, which may be disposed in an electronic device to store at least one instruction or at least one program for implementing a prediction method based on a combined neural network model, where the at least one instruction or the at least one program is loaded and executed by the processor to implement the prediction method based on the combined neural network model provided in the foregoing method embodiment.
Optionally, in an embodiment of the present invention, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and various media capable of storing program codes.
An embodiment of the invention also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions are read by a processor of the computer device from a computer-readable storage medium, and the computer instructions are executed by the processor to cause the computer device to execute the combined neural network model-based prediction method provided in the above-mentioned various alternative embodiments.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And that specific embodiments have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A prediction method based on a combined neural network model, wherein the combined neural network model comprises a plurality of different sub-neural network models, the method comprising:
acquiring image data to be processed;
processing the image data to be processed by utilizing each sub-neural network model respectively to obtain corresponding prediction results;
respectively calculating the similarity between the image data to be processed and the template image corresponding to each sub-neural network model, wherein the template image is determined based on a training image data set of the sub-neural network model corresponding to training;
and based on the similarity between the image data to be processed and each template image, fusing the prediction results corresponding to each sub-neural network model by adopting a weighted average method to obtain a final prediction result.
2. The method according to claim 1, wherein the calculating the similarity between the image data to be processed and the template image corresponding to each sub neural network model respectively comprises:
respectively carrying out non-rigid registration on the image data to be processed and the template image aiming at each template image to obtain a loss value after the registration is finished;
and determining the similarity between the image data to be processed and the template image according to the loss value.
3. The method according to claim 2, wherein the similarity between the image data to be processed and the template image is calculated by the formula:
4. The method according to claim 1, wherein the fusing the prediction results corresponding to the respective sub neural network models by using a weighted average method based on the similarity between the image data to be processed and the respective template images to obtain a final prediction result comprises:
based on a preset weight model, respectively calculating weight coefficients corresponding to prediction results corresponding to the sub-neural network models according to the similarity between the image data to be processed and the template images;
and carrying out weighted average on the prediction results corresponding to the sub-neural network models according to the weight coefficients to obtain the final prediction result.
5. The method according to claim 4, wherein the weight coefficient corresponding to the prediction result corresponding to each of the sub-neural network models is calculated by:
wherein, ω is i Weight coefficient corresponding to prediction result corresponding to ith sub-neural network model, k is number of sub-neural network models, S i For similarity between the image data to be processed and the template image corresponding to the ith sub-neural network model, NS i Is the normalized similarity. />
6. The method of claim 1, further comprising:
determining a training image data set for training each sub-neural network model respectively;
and constructing a template image corresponding to the sub-neural network model according to the training image data set.
7. The method of claim 6, wherein the training image dataset comprises a training dataset and a validation dataset;
the constructing of the template image corresponding to the sub-neural network model according to the training image dataset includes:
constructing a template image corresponding to the sub-neural network model according to the training data set;
or,
and constructing a template image corresponding to the sub-neural network model according to the verification data set.
8. A prediction apparatus based on a combined neural network model, wherein the combined neural network model comprises a plurality of different sub-neural network models, the apparatus comprising:
the acquisition module is used for acquiring image data to be processed;
the processing module is used for processing the image data to be processed by utilizing each sub-neural network model respectively to obtain a corresponding prediction result;
the calculation module is used for calculating the similarity between the image data to be processed and the template images corresponding to the sub neural network models respectively, and the template images are determined based on training image data sets of the sub neural network models corresponding to training;
and the fusion module is used for fusing the prediction results corresponding to the sub-neural network models by adopting a weighted average method based on the similarity between the image data to be processed and each template image to obtain the final prediction result.
9. An electronic device, comprising a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded by the processor and executed to implement the combined neural network model-based prediction method according to any one of claims 1-7.
10. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, and the at least one instruction or the at least one program is loaded by a processor and executed to implement the combined neural network model-based prediction method according to any one of claims 1-7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310148171.8A CN115861662B (en) | 2023-02-22 | 2023-02-22 | Prediction method, device, equipment and medium based on combined neural network model |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310148171.8A CN115861662B (en) | 2023-02-22 | 2023-02-22 | Prediction method, device, equipment and medium based on combined neural network model |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115861662A true CN115861662A (en) | 2023-03-28 |
CN115861662B CN115861662B (en) | 2023-05-12 |
Family
ID=85658640
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202310148171.8A Active CN115861662B (en) | 2023-02-22 | 2023-02-22 | Prediction method, device, equipment and medium based on combined neural network model |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115861662B (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108053885A (en) * | 2017-11-27 | 2018-05-18 | 上海市第六人民医院 | A kind of hemorrhagic conversion forecasting system |
CN113837323A (en) * | 2021-11-08 | 2021-12-24 | 中国联合网络通信集团有限公司 | Satisfaction prediction model training method and device, electronic equipment and storage medium |
US20220084166A1 (en) * | 2019-10-18 | 2022-03-17 | Boe Technology Group Co., Ltd. | Image processing method and device, training method of neural network, image processing method based on combined neural network model, constructing method of combined neural network model, neural network processor, and storage medium |
US20220114389A1 (en) * | 2020-10-09 | 2022-04-14 | GE Precision Healthcare LLC | Systems and methods of automatic medical image labeling |
CN114861941A (en) * | 2022-05-10 | 2022-08-05 | 深延科技(北京)有限公司 | Multi-model fusion method and device, electronic equipment and computer readable storage medium |
-
2023
- 2023-02-22 CN CN202310148171.8A patent/CN115861662B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108053885A (en) * | 2017-11-27 | 2018-05-18 | 上海市第六人民医院 | A kind of hemorrhagic conversion forecasting system |
US20220084166A1 (en) * | 2019-10-18 | 2022-03-17 | Boe Technology Group Co., Ltd. | Image processing method and device, training method of neural network, image processing method based on combined neural network model, constructing method of combined neural network model, neural network processor, and storage medium |
US20220114389A1 (en) * | 2020-10-09 | 2022-04-14 | GE Precision Healthcare LLC | Systems and methods of automatic medical image labeling |
CN113837323A (en) * | 2021-11-08 | 2021-12-24 | 中国联合网络通信集团有限公司 | Satisfaction prediction model training method and device, electronic equipment and storage medium |
CN114861941A (en) * | 2022-05-10 | 2022-08-05 | 深延科技(北京)有限公司 | Multi-model fusion method and device, electronic equipment and computer readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN115861662B (en) | 2023-05-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Yu et al. | Computation offloading for mobile edge computing: A deep learning approach | |
CN110378488B (en) | Client-side change federal training method, device, training terminal and storage medium | |
US20190370530A1 (en) | Facial feature point tracking method, apparatus, storage medium, and device | |
CN112700010B (en) | Feature completion method, device, equipment and storage medium based on federal learning | |
US10560481B2 (en) | Stereotyping for trust management in IoT systems | |
CN108156236A (en) | Service request processing method, device, computer equipment and storage medium | |
CN106919918A (en) | A kind of face tracking method and device | |
Hassani et al. | Context-aware recruitment scheme for opportunistic mobile crowdsensing | |
CN114302421B (en) | Method and device for generating communication network architecture, electronic equipment and medium | |
CN104133765B (en) | The test case sending method of network activity and test case server | |
US20230281513A1 (en) | Data model training method and apparatus | |
CN114302422A (en) | Method and device for processing business by using learning model | |
CN105338537B (en) | A kind of method and terminal of the shared analysis of newly-increased base station site | |
CN110245128A (en) | Method for building up and device, storage medium, the electronic device of meta-model | |
CN113706560A (en) | Ischemia area segmentation method, device, equipment and storage medium | |
CN114492849B (en) | Model updating method and device based on federal learning | |
CN114612531B (en) | Image processing method and device, electronic equipment and storage medium | |
CN115830709A (en) | Action recognition method based on federal edge learning, server and electronic equipment | |
CN113609349A (en) | Rental vehicle inventory detection method, apparatus and medium | |
CN115861662B (en) | Prediction method, device, equipment and medium based on combined neural network model | |
CN115714774A (en) | Calculation force request, calculation force distribution and calculation force execution method, terminal and network side equipment | |
CN116094907B (en) | Complaint information processing method, complaint information processing device and storage medium | |
CN115361295B (en) | TOPSIS-based resource backup method, device, equipment and medium | |
WO2023071616A1 (en) | Service processing method and apparatus, electronic device, and medium | |
CN113487041B (en) | Transverse federal learning method, device and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |