CN114972711B - Improved weak supervision target detection method based on semantic information candidate frame - Google Patents

Abstract

The present invention claims an improved weakly supervised target detection method based on semantic information candidate boxes, which belongs to the field of image processing. The method comprises the following steps: S1: inputting image data to be processed, and adopting preprocessing steps including random horizontal flipping; S2: designing a combined backbone network for fusing features from masked and non-masked network branches; S3: designing a multi-branch detection head network based on a multi-instance selection algorithm; S4: designing a target semantic candidate box, and generating a more reasonable target candidate box by cyclically masking the target semantic information generated by the network model; S5: performing detection on a natural data set. Based on this method, an evaluation is performed on the challenging PASCAL VOC 2007 and 2012 public data sets. Experimental results show that the method proposed in the present invention can achieve good performance on the PASCAL VOC 2007 and 2012 data sets, and outperforms many advanced weakly supervised target detection methods.

Description

An improved weakly supervised object detection method based on semantic information candidate boxes

Technical Field

The present invention belongs to the field of image processing and relates to a weakly supervised target detection method based on a combined backbone network under a semantic information candidate frame combined with a multi-branch detection head network based on a multi-example selection algorithm.

Background Art

Convolutional neural network models have achieved amazing performance in multiple computer vision tasks such as image classification, object detection, and image segmentation. Therefore, in recent years, advanced networks with various advantages have been proposed to further improve the performance of related visual tasks. However, in object detection, most of them are fully supervised methods with a large number of anchor boxes or anchor points, which forces researchers to focus a lot of effort on the accurate labeling of each object coordinate box before training.

Therefore, in order to avoid the need for a large number of real coordinate box labels, a target detection method based on weakly supervised learning has been proposed in recent years, because this method only requires image-level category labels. For weakly supervised target detection models, the model is trained on image-level category labels through the information of all candidate boxes, and after training, reasonable candidate boxes are selected for output, which means that fewer label information resources are required. At present, many weakly supervised target detection models use multi-instance learning and end-to-end model structures to establish a connection between image-level category labels and candidate boxes in the image. For example, as an effective weakly supervised target detection model, the online example classifier fine-tuning method first trains all candidate boxes with limited image-level category information, and then introduces the real target box information of pseudo-image-level categories to further focus on some candidate boxes.

Although the performance of weakly supervised object detection has been greatly improved, these methods still have three problems. First, for the candidate boxes, the quality of the candidate boxes related to the target is poor and the number is small, which hinders the convergence of the model during training. Second, for the features, the target features extracted from the backbone only show a high response in the local target area, which causes the predicted candidate boxes located in this area to have higher credibility than other candidate boxes representing the complete target. Third, for the detection branch, it is unreasonable to have only one pseudo-true target box for each category, because there may be more objects with the same category and different positions in an image.

Summary of the invention

The present invention aims to solve the above problems of the prior art. A weakly supervised target detection method based on semantic information candidate frames is proposed, which is improved from three aspects: the rationality of model feature extraction, the ability to mine potential candidate frames related to the real target frame, and the generation of high-quality target candidate frames before training. The technical solution of the present invention is as follows:

An improved weakly supervised object detection method based on semantic information candidate boxes comprises the following steps:

S1: Input the image data to be processed and use preprocessing steps including random horizontal flipping;

S2: Design a combined backbone network to fuse features from the masked and unmasked network branches. The task of the unmasked network branch is to roughly find the locally significantly different target parts and locate the target, while the task of the masked branch is to shield the significant features and retain the responses of the inconspicuous features in the network.

S3: Design a multi-branch detection head network based on a multi-instance selection algorithm to generate more pseudo-real target boxes with higher confidence for each target category, so that the network model can obtain more candidate boxes related to the target and obtain more reasonable training.

S4: Design target semantic candidate boxes, and generate more reasonable target candidate boxes by cyclically masking the target semantic information generated by the multi-branch detection head network model;

S5: Testing on natural datasets.

Furthermore, the step S1 inputs the image data to be processed and adopts a preprocessing step including random horizontal flipping, which specifically includes the following steps:

The network model is trained by randomly adjusting the size of the shortest side of the training images using five image scales {480, 576, 688, 864, 1200} through random horizontal flipping operations; in the test phase, the predicted output results are calculated by comprehensively analyzing all scales of each image and its horizontal flipping.

Furthermore, the predicted output result is calculated by comprehensively analyzing all scales of each image and its horizontal flipping, specifically including:

The scale of each image is set in order of 480, 576, 688, 864 and 1200. The images of 5 different scales and their horizontal flipping are input into the network model to obtain the output results of the images of different scales and their horizontal flipping in the network model. Finally, the prediction results of images of different scales are scaled to the results of the same scale, and the scaled output results are screened using the post-processing algorithm to obtain the final prediction results.

By calculating the IoU between the sample prediction result and the real target frame coordinates, first, determine the sample attributes according to the preset threshold (0.5 or 0.75). Then, sort all samples from high to low according to their classification results. Traverse the sorted samples, and calculate the precision and recall of the traversed samples according to formula (1) and formula (2).

Among them, TP, FP and FN represent true positives, false positives and false negatives.

Based on the Precision and Recall obtained from each traversal, a curve with Recall as the X-axis and Precision as the Y-axis is constructed. Finally, the average precision (AP) is obtained by calculating the area of the curve enclosed by Precision and Recall, and the mean average precision (mAP) is obtained by calculating the AP of each category.

Furthermore, the step S2 further includes the following steps:

Given an image, candidate boxes are generated based on pixel-level and target semantic features and input into the spatial pyramid pooling layer SPP. After extracting features and coordinates of candidate boxes from the combined backbone network, the candidate boxes are mapped to the feature map to obtain the region of interest RoI. Since each RoI has its own size, the SPP layer obtains a RoI with a fixed size by replacing the maximum pooling layer located at the last layer of the combined backbone network. Then, the parallel fully connected layer in the improved detection head network will output a series of feature vectors of candidate boxes.

Furthermore, in step S2, the method for dividing the significant and inconspicuous features is to extract the average and maximum values from the feature map before passing through the two independent branches; if the feature value of a specific position is higher than the threshold, the significant feature will be shielded, so that the response of the inconspicuous feature after passing through the combined backbone network is enhanced; in addition, there are two methods for shielding the significant features; one is a hard masking method, and the other is a soft masking method, which can be shown in formula (3) and formula (4) respectively,

Among them, fi _,j represents the feature value of a specific position, mean, Max, respectively represent the mean and maximum value of the current feature map, and Hard masking and soft masking methods are used to represent the results respectively, and α is a hyperparameter used to control the threshold size.

Furthermore, in S3, the multi-branch detection head network includes the following two steps:

First, all candidate boxes are jointly trained in the top branch of the multi-branch detection head network to find potential target candidate boxes, and the image-level category label is used as the label of this branch; then, the network model only focuses on the potential candidate box area related to the target, and trains the network part associated with this area, and the pseudo-real target detection box label generated in the previous network branch is used as the new monitoring input to the next network branch.

Furthermore, a multi-example selection algorithm is introduced in step S3 to generate multiple pseudo-real coordinate frames with high confidence in each category and send them to the next network branch so as to train candidate frames that are more relevant to the real target; based on the different scores of the candidate frames in each category in each network branch, all scores are sorted from high to low, and the focus is placed on the candidate frames P′ with the top 10% scores as the source of pseudo-real target detection frames; a threshold of K is set to limit the number of pseudo-real target detection frames in each category; then, the same threshold is used in each category to determine whether the candidate frame can serve as a pseudo-real target frame box.

Furthermore, in the improved detection head network, each class will generate multiple pseudo-real target detection boxes. The candidate boxes with the maximum IoU calculated in these pseudo-real target boxes above and below 0.5 will be regarded as foreground and background, respectively. The standard cross entropy loss is multiplied by the γ ^k weight in the k-th detection branch to balance the loss function of the foreground and background samples by adjusting the value of γ ^k . For the k-th improved selection detection branch, its loss function is as follows:

in, They represent the prediction results of the pseudo-real target box related to the current candidate box, the background category label information, and the output results of the background category samples. γ ^k represents the loss value and positive and negative sample balance weight of the k-th fine-tuned detection head. and They represent the background cost of the cth category and the pth candidate box respectively. C is the total number of categories in units of the kth category. |P| is equal to the total number of candidate boxes. C+1 represents the category label of the background. and Represents the label information and output results of the cth category of the candidate box p in the kth detection branch;

Then, the standard binary cross entropy function of the top coarse selection detection head, Formula (8), is incorporated into the entire model loss function so that the network can be jointly trained; therefore, the loss function used to train the entire network is shown in Formula (9);

P _c , y _c , and L represent the sum of the prediction results of all candidate boxes in the cth class, the sample label information of the cth class, and the total loss value of the network model, respectively.

Furthermore, in step S4, a circular masking method is used to mask these salient feature parts, thereby generating a reasonable target candidate frame, which specifically includes:

The whole process of generating the target semantic candidate box can be divided into three stages, namely the shielding stage, the synthesis stage and the generation stage. The purpose of the shielding stage is to eliminate the local salient features of the object.

The merging stage is to fuse these network responses generated on average in each cycle;

The generation stage generates candidate boxes based on the generated target area.

The advantages and beneficial effects of the present invention are as follows:

The innovation of the present invention is mainly the combination of claims 2-9. The present invention proposes a weakly supervised target detection method based on semantic information candidate frames, which realizes the detection of image objects in a weakly supervised learning manner. First, a candidate frame based on target semantic information generated by a cyclic masking method is proposed. This method can improve the number and quality of candidate frames related to the target and help the network model converge better during training; then a dual-branch backbone network structure that integrates the original feature branch and the significant feature suppression branch is proposed, called a combined backbone network, to improve the network's response to inconspicuous features and avoid the detection results being dominated by the local significant features of the target; finally, a multi-branch detection head network based on a multi-instance selection algorithm is proposed to generate more pseudo-real target frames with high confidence for each category, so that the model can be trained in a more reasonable way. Therefore, the present invention improves the model from three aspects, namely, the target candidate frame, the backbone network, and the multi-branch detection head network, which can respectively solve the problem of insufficient number and poor quality of candidate frames related to the target, the feature response of the local significant features dominating the network, and the lack of pseudo-real target frames causing deviations in the candidate frame label information, thereby acting as a whole to improve the detection accuracy of the model for the target in weakly supervised learning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing three core modules and an overall flow chart of a preferred embodiment of the present invention.

Figure 2 shows the difference between hard mask and soft mask methods.

Figure 3 shows the statistical results of the relative position between the center of the target area and the center of the corresponding true target box on the PASCAL VOC 2007 test set.

FIG4 is a flowchart of generating a target semantic candidate box.

Figure 5 shows the comparison between the target semantic candidate boxes without and with random constraint functions.

DETAILED DESCRIPTION

The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

The technical solution of the present invention to solve the above technical problems is:

A method for classifying and locating image threat objects based on multi-attention and semantics, the method comprising the following steps:

S1: Data preprocessing;

S2: Design a combined backbone network to enhance the response of inconspicuous target features;

S3: Design a multi-branch detection head network based on a multi-instance selection algorithm to facilitate the training of more potential target candidate boxes;

S4: Design object semantic candidate boxes to improve the quality and quantity of object candidate boxes in images;

S5: Detection on the natural datasets PASCAL VOC 2007 and VOC 2012;

Optionally, the S1 specifically includes the following steps:

The network model is trained by randomly resizing the shortest side of the training images using five image scales {480, 576, 688, 864, 1200} through random horizontal flipping operations. During the test phase, the predicted output is calculated by comprehensively analyzing all scales of each image and its horizontal flipping.

In S2, in order to prevent the network from overfitting the local salient area of the target, a combined backbone network is designed to fuse the features from the masked and non-masked network branches. The task of the non-masked network branch is to roughly find the target part with significant local differences and locate the target, while the task of the masked branch is to shield the significant features and retain the response of the inconspicuous features in the network. After combining these two independent branches, the response of the inconspicuous features can be effectively improved, and the detection results can be prevented from being dominated by the local obvious target features.

Preferably, in S3, in order to parse multiple pseudo-real target boxes generated in each category, the present invention proposes a multi-branch detection head network based on a multi-instance selection algorithm to find more pseudo-real target boxes with high confidence, so that more potential target candidate boxes can be trained.

Preferably, in S4, in order to improve the quality and quantity of candidate boxes, the present invention proposes a method for generating candidate boxes based on target semantic features rather than pixel-level image information (such as Selective Search or Edge Box algorithm) to further help the network converge better during training. The method generates more reasonable target candidate boxes by performing cyclic masking on the target semantic information generated by the network model, thereby improving the quantity and quality of the target candidate boxes and helping the detection network model of the present invention to converge better during training.

Preferably, in said S5, during the training process, the present invention uses Adam as the optimizer of the network model, and the initial learning rate is 1×10 ^-5 . The minimum batch size set by the present invention is set to 8 and 16 input images in PASCAL VOC 2007 and PASCAL VOC 2012, respectively. For these two data sets, the present invention performs 30k iterations of training on the model, and the learning rate is reduced to 10% of the original after 20k iterations of training.

Optionally, in S2, given an image, a candidate box is generated based on pixel-level and target semantic features and input into a spatial pyramid pooling layer (SPP). Specifically, after extracting features and coordinates of candidate boxes from the combined backbone network, the present invention maps the candidate boxes to feature maps to obtain a region of interest (RoI). Since each RoI has its own size, the SPP layer obtains a RoI with a fixed size by replacing the maximum pooling layer located at the last layer of the combined backbone network. Then, the parallel fully connected layer in the improved detection head network will output a series of feature vectors of candidate boxes.

To further reduce the overfitting of local obvious features, a combined backbone network structure is designed to improve the response of the inconspicuous feature area. Specifically, after extracting the underlying features, the network will be divided into two independent branches. The first branch is to find the salient features of the object to roughly locate the position, and the second branch is to cover these salient features but retain the inconspicuous feature responses.

The method of dividing the significant and insignificant features is to extract the average and maximum values from the feature map before passing through the two independent branches. If the value is higher than the threshold, the significant feature will be masked, so that the response of the insignificant feature after passing through the combined backbone network is enhanced. In addition, there are two methods to mask the significant features. One is the hard masking method and the other is the soft masking method, which can be shown in formula (1) and formula (2) respectively.

Among them, _fi,j represents the eigenvalue at a specific position and The hard masking and soft masking methods are used to represent the results respectively. α is a hyperparameter used to control the threshold size.

Compared with the hard mask method, the soft mask method has two advantages. First, it can avoid the situation where some features with values slightly above the threshold cannot be enhanced, because the hard mask function will set them to zero in the shielding branch. Second, if there is too much background information in the image, the global average eigenvalue will be reduced, and the mask branch of the hard mask method can only focus on features with lower eigenvalues. Therefore, the soft mask method can still work well when a small target is located in a large background.

Optionally, in S3, the workflow of the multi-branch detection head network can be summarized into the following two steps. First, all candidate boxes are trained together in the top branch part of the network to find potential target candidate boxes. The image-level category label is used as the label of the branch. Then, the network model only focuses on the potential candidate box area related to the target, and trains the network part associated with the area, and the pseudo-real target detection box label generated in the previous network branch is used as the new monitoring input to the next network branch.

The network will introduce a Multiple Instance Selection (MIS) algorithm, which generates multiple pseudo-real coordinate boxes with high confidence in each category and sends them to the next network branch in order to train candidate boxes that are more relevant to the real target. Based on the different scores of the candidate boxes in each category in each network branch, the present invention sorts all scores from high to low and focuses on the candidate boxes P' with the top 10% scores as the source of pseudo-real target detection boxes. A threshold of K will be set to limit the number of pseudo-real target detection boxes in each category. Then, the present invention uses a method similar to non-maximum suppression (NMS) to use the same threshold in each category to determine whether the candidate box can serve as a pseudo-real target box. Specifically, for the i-th candidate box in P', if the IOU between it and the previously generated pseudo-real target box is lower than the threshold t, the candidate box will be appended to the pseudo-real target box list In addition, since the network parts corresponding to these candidate box areas may be trained in other categories, the scores of these selected candidate boxes will be reset to zero. Finally, when generating pseudo-real target boxes Previously, the sequence of class vectors c was randomized to reduce the impact that classes with priority can select more candidate boxes.

In the improved detection head network, multiple pseudo-real target detection frames will be generated for each class. Candidate frames with maximum IoU above and below 0.5 calculated in these pseudo-real target frames will be regarded as foreground and background, respectively. In order to train the detection network model of the present invention more effectively, it is necessary to balance the positive and negative candidate frames, because the number of backgrounds far exceeds that of foregrounds. Therefore, the present invention multiplies the standard cross entropy loss by the γ ^k weight in the k-th detection branch so as to balance the loss function of the foreground and background samples by adjusting the value of γ ^k . In order to avoid overfitting the model by this parameter, all fine branches have the same γ adjustment. The experiments of the present invention show that this weight can significantly improve the model performance. In addition, let w _p be equal to the result of the closest pseudo-real target frame of the candidate frame p, because at the beginning of training, the results of these pseudo-real target frames in the corresponding classes are not reliable enough, and w _p results are small at the beginning of training. This variable can prevent the model proposed by the present invention from being disturbed by these unreliable pseudo-real target frames. Therefore, for the k-th improved selection detection branch, its loss function is as follows.

in, and They represent the background cost of the cth category and the pth candidate box respectively. C is the total number of categories in units of the kth category. |P| is equal to the total number of candidate boxes. C+1 represents the category label of the background. and Represents the label information and output results of the cth category of the candidate box p in the kth detection branch.

Then, the standard binary cross entropy function of the top coarse selection detection head (as shown in Equation (6)) is incorporated into the overall model loss function so that the network can be trained jointly. Therefore, the loss function used to train the entire network is shown in Equation (7).

Optionally, in S4, for the improved candidate frame module, although the Selective Search method can generate a large number of candidate frames, the number of background candidate frames far exceeds the number of target candidate frames. In addition, many candidate frames related to the target usually only contain a part of the target. In order to solve these problems, the present invention improves the number and quality of candidate frames related to the target according to the semantic features of the target.

The target semantic candidate box is generated by analyzing the target semantic feature responses generated by the model that already has a certain knowledge representation capability. Gradient Class Activation Mapping (Grad CAM) is a visualization algorithm designed to display the response of target features with a specific category in the network. Specifically, Grad CAM can use the back-propagation gradient of each category and adopt the global average pooling function to calculate the average weight of each feature map so that the model outputs the target response. These average weights are then multiplied by the corresponding values of the feature map before the ReLU function. The whole process can be summarized as follows:

Among them, y _c is the output of category c, Represents the value of a specific position in the kth feature map, and Z represents the area of the feature map is the average response of the ki-th functional map, and L _c is the activation map of the specific c-th category. The candidate box generation method proposed in the present invention is characterized in that no retraining is required, because VGG16 pre-trained on ImageNet can be used as the backbone network for generating target semantic features, which means that the method of the present invention is more general and can be used for data sets with more categories. In addition, the method proposed in the present invention can also be based on the cyclic mask method proposed in the present invention and the generation of the target semantic candidate box, and a more reasonable candidate box can be obtained by using fewer manual thresholds.

Since high responses are always dominated by salient features, it is important to mask these areas before generating feasible candidate boxes. Therefore, the present invention adopts a cyclic masking method to mask these salient feature parts and then generate reasonable target candidate boxes.

The whole process of generating the target semantic candidate box can be divided into three stages, namely the masking stage, the synthesis stage and the generation stage. First, the purpose of the masking stage is to eliminate the local salient feature parts of the object. When the original image passes through the neural network, the output will have a score result related to the corresponding category. Before calculating the gradient by back propagation, the present invention selects the class corresponding to the top 5 scores as the response source and generates the network response before entering the dynamic binary threshold function. Then, if the response generated at a specific location is higher than the threshold of the dynamic function, the salient features of the target will be masked in the original image. After masking these obvious feature areas, the masked image will replace the previous image and serve as the input of the network in the current cycle.

The number of cycles set in the present invention is 2, because the first cycle is to find the most discriminative part of the object, and the second cycle is to find the secondary area. More cycles may be effective, but the present invention finds that the effect is not obvious and time-consuming, because the above steps have already detected most of the areas related to the target. For the dynamic binary threshold function, the threshold size follows formula (10), where i = 1, 2, ..., T, _Ti represents the threshold of the i-th cycle.

T _i = 100 + 20 * (i-1) (10)

Based on a lot of empirical observations of images, the present invention found that low thresholds lead to oversized masks containing multiple targets. 100 is a suitable benchmark for generating reasonable masks in the first stage. Since the previous cycle removes the distinguishing part, it means that when processing the mask image in the network, the remaining target area is smaller and the network is more sensitive to noise. Therefore, using a higher threshold in the next cycle can avoid the network being disturbed by noise such as background.

The merging stage is to fuse these network responses generated on average in each cycle. The comprehensive feature response should generate multiple sets of binary target regions as the basis of candidate boxes through a fixed threshold. In addition, since small targets may exist in large target regions, increasing the threshold can generate small target regions contained in large target regions, so that candidate boxes related to these small targets can be generated. However, a compromise candidate box is that a small interval will result in unclear differences in each region, while a large interval will result in the generated region representing only part of the same target.

The generation stage generates candidate boxes based on the generated target regions. However, the present invention cannot obtain candidate boxes only from the edges of the target regions. The difference between adjacent target regions is the main factor in generating target semantic candidate boxes in the present invention. The coordinates of the generated candidate boxes can be calculated as follows:

Where P _right,i , P _left,i , P _top,i and P _bottom,i represent the right, left, top and bottom coordinates of the i-th candidate box, respectively. R _right , R _left , R _top and R _bottom represent the coordinates of the target region generated previously. N is a constant representing the number of candidate boxes per region, and t is a threshold to determine the spacing distance limit factor. Δ _right , Δ _left , Δ _top and Δ _bottom are the differences calculated at the corresponding positions.

According to the statistical results on the PASCAL VOC 2007 test set, _SR and _SGT represent the area of the generated target region and the area of its associated true target box, respectively. It can be noted that although the target region has a high correlation with the true target box (i.e., IoU(R,GT)>0.5), there is still a 62.7% confidence level to determine that the area of the generated target region is smaller than the area of the associated true target box. Therefore, the present invention needs to expand the generated target region to obtain a reasonable target semantic candidate box.

The valid target region should have a high correlation with the true coordinate frame or the inside of the true coordinate frame. If the valid target region is at the top or left of the true target frame, other regions must be at the bottom or right. In weakly supervised target detection, there is no example-level label information. However, according to the statistical results obtained by the present invention, no matter where the valid target region is in the image, there is at least a 70% confidence probability that the center of the true target frame is closer to the center of the image, which means that it will be more reasonable to generate a target semantic candidate frame toward the center of the image. Therefore, if a valid target region is at the top or left of the center of the image, more attention is focused on expanding the bottom or right side of the target generation region, and it is more likely to reduce the distance between the center of the target region and the center of the true coordinate frame.

In addition, in order to reduce the impact of the target area larger than the true target box and bring irreversible information, the random restriction function f in formula (15) is incorporated into formulas (11)-(14) to reduce the growth of the coordinates, thereby reducing the introduction of irrelevant information. In addition, the target located at the edge of the image will cause an imbalance in the interval when calculating the coordinates of the target semantic candidate box. For example, if the target area is located in the upper left corner of the image, the results of Δ _top and Δ _right will be much larger than Δ _bottom and Δ _left . Therefore, there will be a situation where there is too much background information including the lower right corner area. The random constraint function therefore takes this situation into account and brings a relatively strong constraint on the growth of the lower right interval. However, since most target areas are smaller than their associated true target boxes, the stronger restriction parameters in formula (15) will result in the generation of more candidate boxes with smaller areas.

As described above, adjacent target regions have the result of determining Δ. The previous target region will become the boundary of the next target region. But for the first target region, the size of the original image is used as its farthest boundary. Since the small target region inside the large region sometimes only represents a part of the target, rather than other small targets, this means that these regions should not be considered when generating target semantic candidate frames. In order to reduce the impact, the present invention calculates the IoU between the previous target region and the next target region to decide whether to generate candidate frames based on these smaller regions.

In addition, when CNN calculates each local feature within the local receptive field, an image pyramid is introduced, which means that the response will change depending on the scale of the target. Therefore, the present invention sets s = [1, 0.75, 0.5] to further generate more reasonable candidate boxes for small, medium and large-scale targets. The initial target area threshold is equal to 100. In order to generate more target area sets, each lifting threshold after the function set by the present invention will be higher than 30 of the previous threshold, that is, a fixed binary threshold list T _fix = [100, 130, 160] is obtained. In addition, a fixed number of candidate boxes will be generated for each target area. According to formulas (11)-(14), when calculating Δ, the interval distance is determined by N. Here, the present invention only selects the first 10 candidate boxes (i.e., i∈[0, 9]) from each target area, and sets N of the first group and the remaining two groups of target areas to 50 and 20, respectively. Finally, the threshold t set by the present invention in formula (8) is 8.

As shown in Figure 1, an example of the present invention provides a weakly supervised target detection method based on a combined backbone network under a semantic information candidate box combined with a multi-branch detection head network based on a multi-instance selection algorithm. The method can be implemented by a terminal or a server, and the method includes: the first module is to combine pixel-level and object semantic candidate boxes, and we define this module as an improved candidate box. The second module is to reduce the overfitting phenomenon of the network to significant features by improving the response of unobvious object features. We call this module a combined backbone network. The last one is a multi-branch detection head network module of a multi-instance selection algorithm, and its purpose is to generate more pseudo-real target boxes to obtain more potential candidate boxes related to the target.

The difference between hard masking and soft masking is shown in Figure 2, where we assume that α is equal to 0. After the soft masking method, the eigenvalues around the object are the same, which means that all features around the object have the same importance. However, since the average eigenvalue reduces the background area too much, the hard masking method cannot improve the response of non-salient features.

The statistical results of the relative position between the center of the target region and the center of the relevant true target box on the PASCAL VOC 2007 test set are shown in Figure 3. It can be observed that no matter where the valid target region is in the image, there is at least 70% confidence that the center of the true target box is closer to the center of the image, which means that it is more reasonable to generate a target semantic candidate box towards the center of the image.

As shown in Figure 4, the target semantic candidate box generation flow chart, using Grad CAM, can generate responses to objects from the network. The purpose of the dynamic threshold function is to generate a mask to mask obvious features. The previous image will be replaced by the masked image, and then the process will start again. When the number of loops reaches the preset value of the total loop, multiple responses are merged to display the comprehensive response result. Then, the fixed threshold function filters out noise and generates the target area. Increasing the threshold will make the area change. Finally, candidate boxes are generated based on these target areas and formulas (11)-(15).

Comparison between the target semantic candidate boxes without random constraint function and with random constraint function as shown in Figure 5. The orange and blue boxes represent the true target box and the generated target semantic candidate box, respectively. The generated candidate box with random constraint function will be relatively compact with the true target box, which means that the background information in the generated candidate box can be reduced.

In order to verify the method based on multi-attention mechanism and high semantic and high resolution feature combined network provided by the example of the present invention, for multiple hyperparameters of the model, γ ^k is equal to 0.7, which represents the weight of the target candidate box in the loss function of each fine-tuning detection branch; α in the combined backbone network is set to 0.1. For the improved pair selection detection head, K and threshold t are set to 5 and 0.05 by default. The number of fine-tuning detection branches is set to 3. Through random horizontal flipping operations, the present invention uses five image scales {480, 576, 688, 864, 1200} to randomly adjust the size of the shortest side of the training image. In the test phase, the output is calculated by comprehensively analyzing all scales of each image and its horizontal flipping.

The present invention uses the Selective Search algorithm to generate pixel-level candidate boxes, and combines the target semantic candidate boxes of the present invention as example samples of the network. VGG16 (without batch normalization) pre-trained on the ImageNet dataset is used as the backbone of feature extraction. In addition, the penultimate maximum pooling layer and the subsequent convolutional layers are changed to hole convolutional layers to expand the receptive field. For the newly added layers, a Gaussian distribution is used for initialization with a mean of 0 and a standard deviation of 0.01. Finally, our experiments are implemented in the PyTorch deep learning framework, Python, and C++. All experiments are run on NVIDIA GTX Titan V GPU and Intel Xeon Silver 4110 CPU (2.10 GHz).

Experimental Results

In this example, AP and mAP are used to evaluate the model performance at an IoU threshold of 0.5. In addition, the CorLoc evaluation indicator is used to evaluate the positioning accuracy of the model on the trainval set and the test set.

Table 1 Detection results of each category using VGG16 on PASCAL VOC 2007

Table 2 Detection results of each category using VGG16 on PASCAL VOC 2012

Table 3 Correct positioning results for each category using VGG16 on PASCAL VOC 2007

Table 4 Correct positioning results for each category using VGG16 on PASCAL VOC 2012

It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims (2)

1. An improved weakly supervised object detection method based on semantic information candidate boxes, characterized in that it includes the following steps: S1: Input the image data to be processed and use preprocessing steps including random horizontal flipping; S2: Design a combined backbone network to fuse features from the masked and unmasked network branches. The task of the unmasked network branch is to roughly find the locally significantly different target parts and locate the target, while the task of the masked branch is to shield the significant features and retain the responses of the inconspicuous features in the network. S3: Design a multi-branch detection head network based on the multi-instance selection algorithm. The multi-branch detection head network based on the multi-instance selection algorithm generates more pseudo-real target boxes with higher confidence for each target category, so that the network model can obtain more candidate boxes related to the target, and thus obtain more reasonable training; S4: Design target semantic candidate boxes, and generate more reasonable target candidate boxes by cyclically masking the target semantic information generated by the multi-branch detection head network model; S5: Testing on natural datasets; The step S1 inputs the image data to be processed and adopts a preprocessing step including random horizontal flipping, which specifically includes the following steps: The network model is trained by randomly adjusting the size of the shortest side of the training images using five image scales {480, 576, 688, 864, 1200} through random horizontal flipping operations; in the test phase, the predicted output results are calculated by comprehensively analyzing all scales of each image and its horizontal flipping; The S2 further includes the following steps: Given an image, candidate boxes are generated based on pixel-level and target semantic features and input into the spatial pyramid pooling layer SPP. After extracting features and coordinates of candidate boxes from the combined backbone network, the candidate boxes are mapped to the feature map to obtain the region of interest RoI. Since each RoI has its own size, the SPP layer obtains a RoI with a fixed size by replacing the maximum pooling layer at the last layer of the combined backbone network. Then, the parallel fully connected layer in the improved detection head network will output a series of feature vectors of candidate boxes. In step S2, the method of dividing the significant and inconspicuous features is to extract the average value and the maximum value from the feature map before passing through the two independent branches; if the feature value of a specific position is higher than the threshold, the significant feature will be shielded, so that the response of the inconspicuous feature after passing through the combined backbone network is enhanced; in addition, there are two methods to shield the significant features; one is a hard masking method, and the other is a soft masking method, which can be shown in formula (3) and formula (4) respectively, Among them, fi _,j represents the feature value at a specific position, mean and Max represent the mean and maximum value of the current feature map respectively. and The hard masking and soft masking methods are used to represent the results. α is a hyperparameter used to control the threshold value. In S3, the multi-branch detection head network includes the following two steps: First, all candidate boxes are jointly trained in the top branch of the multi-branch detection head network to find potential target candidate boxes, and the image-level category label is used as the label of this branch; then, the network model only focuses on the potential candidate box area related to the target, and trains the network part associated with this area, and the pseudo-real target detection box label generated in the previous network branch is used as the new monitoring input to the next network branch; In the step S3, a multi-example selection MIS algorithm is introduced to generate multiple pseudo-real coordinate frames with high confidence in each category, and send them to the next network branch to train candidate frames that are more relevant to the real target; based on the different scores of the candidate frames in each category in each network branch, all scores are sorted from high to low, and the focus is placed on the candidate frames P′ with the top 10% scores as the source of pseudo-real target detection frames; a threshold of K is set to limit the number of pseudo-real target detection frames in each category; then, the same threshold is used in each category to determine whether the candidate frame can serve as a pseudo-real target frame box; In the improved detection head network, each class will generate multiple pseudo-real target detection boxes. The candidate boxes with the maximum IoU calculated in these pseudo-real target boxes above and below 0.5 will be regarded as foreground and background respectively. The standard cross entropy loss is multiplied by the γ ^k weight in the k-th detection branch to balance the loss function of the foreground and background samples by adjusting the value of γ ^k . For the k-th improved selection detection branch, its loss function is as follows: in, They represent the prediction results of the pseudo-real target box related to the current candidate box, the background category label information, and the output results of the background category samples. γ ^k represents the loss value and positive and negative sample balance weight of the kth fine-tuned detection head, and They represent the background cost of the cth category and the pth candidate box respectively. C is the total number of categories in units of the kth category. |P| is equal to the total number of candidate boxes. C+1 represents the category label of the background. and Represents the label information and output results of the cth category of the candidate box p in the kth detection branch; represents the prediction results of the pseudo-real target box related to the current candidate box, the background category label information and the prediction results of the background category samples; Then, the standard binary cross entropy function of the top coarse selection detection head, Formula (8), is incorporated into the entire model loss function so that the network can be jointly trained; therefore, the loss function used to train the entire network is shown in Formula (9); P _c , y _c , and L represent the sum of the prediction results of all candidate boxes in the cth class, the sample label information of the cth class, and the total loss value of the network model respectively; In step S4, a circular masking method is used to mask these salient feature parts, thereby generating a reasonable target candidate frame, which specifically includes: The whole process of generating the target semantic candidate box can be divided into three stages, namely the shielding stage, the synthesis stage and the generation stage. The purpose of the shielding stage is to eliminate the local salient features of the object. The merging stage is to fuse these network responses generated on average in each cycle; The generation stage generates candidate boxes based on the generated target area. 2. According to claim 1, the improved weakly supervised object detection method based on semantic information candidate box is characterized in that the predicted output result is calculated by comprehensively analyzing all scales of each image and its horizontal flipping, specifically comprising: The scale of each image is set in order according to the scales of 480, 576, 688, 864 and 1200; the images of 5 different scales and their horizontal flipping are input into the network model to obtain the output results of the images of different scales and their horizontal flipping in the network model; finally, the prediction results of the images of different scales are scaled to the results of the same scale, and the scaled output results are screened by the post-processing algorithm to obtain the final prediction results; By calculating the IoU size between the sample prediction result and the real target frame coordinates, first, determine the sample attribute according to the preset threshold (0.5 or 0.75); then, sort all samples from high to low according to their classification results; traverse the sorted samples, and calculate the precision and recall of the traversed samples according to formula (1) and formula (2); Among them, TP, FP and FN represent true positives, false positives and false negatives; Based on the Precision and Recall obtained from each traversal, a curve with Recall as the X-axis and Precision as the Y-axis is constructed; finally, the average precision AP is obtained by calculating the area of the curve enclosed by Precision and Recall, and the mean average precision mAP is obtained by calculating the AP of each category.