CN113506316A - Method and device for segmenting video object and network model training method - Google Patents

Abstract

The embodiments of the present application provide a method, a device, and a network model training method for segmenting a video object. The method for segmenting a video object includes: extracting features of at least one historical frame image before the current frame image, and obtaining the at least one historical frame image feature pairs of each historical frame image in the image; extract the features of the current frame image to obtain the feature pair of the current frame image; extract the features of the current frame image to obtain the feature pair of the current frame; The feature pair of the current frame image, the feature pair of the current frame image, and the decoder, to obtain the segmentation mask of the target of interest in the current frame image; wherein, each historical frame image in the at least one historical frame image is the The previous frame or images of the current frame image, the feature pair includes a key matrix and a value matrix. Some embodiments of the present application implement inter-frame tracking through an enhanced short-term memory network, which significantly improves the video object segmentation accuracy for the current frame image.

Description

Method, device and network model training method for segmenting video objects

technical field

The present application relates to the field of video object segmentation, and in particular, the embodiments of the present application relate to a method, an apparatus, and a network model training method for segmenting video objects.

Background technique

Video object segmentation is an important subject in computer vision, so video object segmentation is widely used in video surveillance, object tracking and mobile phone video processing. Video object segmentation includes two parts: image segmentation and object tracking. The so-called video object tracking refers to accurately segmenting all objects in subsequent video frames for one or more predefined objects in a given video sequence.

Existing video object segmentation is mainly limited by segmentation accuracy. On the one hand, there are many problems in the accuracy of object segmentation in a single frame of images. In a video sequence, the same object has very large changes in posture and shape in different frames, which often have very large differences with the predefined posture and shape. On the other hand, when there are multiple objects of the same type in the video, video segmentation needs to distinguish the object from other objects, instead of It will be mistakenly segmented into other similar objects. For these reasons, the accuracy of existing video object segmentation techniques is greatly limited.

Therefore, how to improve the accuracy of video object segmentation has become an urgent technical problem to be solved.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide a method, device and network model training method for segmenting video objects. The new deep learning-based methods provided by some embodiments of the present application are based on an enhanced short-term memory network, and a pair of The network that combines the features of the current frame image and the cropped image features integrates object segmentation and tracking into a neural network, and supervises the segmentation results of the current frame image through the segmentation results of the historical frames, which achieves a significant improvement in the accuracy of video object segmentation.

In a first aspect, some embodiments of the present application provide a method for segmenting a video object, the method comprising: extracting features of at least one historical frame image before the current frame image, and obtaining each of the at least one historical frame image feature pairs of historical frame images; extract features of the current frame images to obtain feature pairs of the current frame images; obtain feature pairs of the historical frame images, feature pairs of the current frame images and a decoder, obtain The segmentation mask of the object of interest in the current frame image; wherein the feature pair includes a key matrix and a value matrix.

Some embodiments of the present application implement inter-frame tracking by extracting features of at least one historical frame image adjacent to the current frame image through an enhanced short-term memory network, which significantly improves the accuracy of video object segmentation for the current frame image.

In some embodiments, the feature of the current frame image is extracted by a current frame encoder, and the current frame encoder includes a convolution layer, a down-sampling layer, and a feature similarity fusion module; wherein, the current frame encoder is used. Extracting the feature of the current frame image to obtain the feature pair of the current frame image, comprising: using the convolution layer and the downsampling layer to extract the feature of the current frame image to obtain the current frame feature map; using the The convolutional layer and the downsampling layer extract the features of the cropped image to obtain a cropped image feature map, wherein the cropped image is obtained by cropping the target of interest from the first frame of image, and the first frame of image is the frame in which the target of interest appears for the first time in the video sequence; according to the feature similarity fusion module, the current frame feature map and the cropped image feature map are fused to obtain a fused image, and based on the fused image to obtain The feature pair of the current frame image.

Some embodiments of the present application better highlight the current target of interest that needs to be segmented and further improve the segmentation accuracy by performing similarity fusion between the extracted current frame image and the feature map of the cropped image containing the target of interest.

In some embodiments, the feature similarity fusion module fuses the current frame feature map and the cropped image feature map through a short-term memory network to obtain the fused image.

Some embodiments of the present application provide a network structure for fusing the extracted feature map of the current frame and the feature map of the cropped image, and the feature fusion of the current frame image and the cropped image can be realized through the short-term memory network, which better highlights the target of interest.

In some embodiments, the feature similarity fusion module obtains the fusion image according to the following formula:

表征所述融合图像对应的矩阵，feat_t表征与所述当前特征图对应的矩阵，feat_p表征与所述裁剪图像特征图对应的矩阵。in,

represents the matrix corresponding to the fusion image, feat _t represents the matrix corresponding to the current feature map, and feat _p represents the matrix corresponding to the cropped image feature map.

Some embodiments of the present application provide a calculation formula for fusing the extracted features of the current frame image and the extracted features of the cropped image, so as to realize the quantitative representation of the fused features.

In some embodiments, the feature of the current frame image is extracted through an enhanced short-term memory network, the enhanced short-term memory network includes at least one encoder of a semantic segmentation network, and each encoder in the at least one encoder connected in parallel, and each encoder receives an input historical frame image and a segmentation result of the historical frame image.

Some embodiments of the present application mine features of multiple frames of historical images by connecting multiple encoders in parallel, thereby better realizing inter-frame tracking.

In some embodiments, obtaining the segmentation mask of the object of interest in the current frame image according to the feature pair of the historical frame images, the feature pair of the current frame image, and the decoder includes: The feature of the current frame image is respectively fused to the included current key matrix and the feature of each historical frame image to the included historical key matrix, to obtain a fusion key matrix; the fusion key matrix and the current frame image The current value matrix included in the feature pair is input to the decoder to obtain the segmentation mask.

Some embodiments of the present application provide a method for fusing the features of the mined historical frame image and the features of the mined current frame image, and the fused feature pair (that is, the fusion key matrix and the current frame image) obtained by this fusion method The feature pair includes the current value matrix) to improve the accuracy of the segmentation mask decoded by the decoder.

In some embodiments, performing a fusion operation on the current key matrix included in the feature pair of the current frame image and the historical key matrix included in the feature pair of each historical frame image, respectively, to obtain a fusion key matrix, including: respectively: Acquire the first correlation degree of each historical key matrix in the current key matrix and all the historical key matrices; obtain the fusion key matrix according to the first correlation degree.

In some embodiments of the present application, the fusion key matrix is determined by calculating the correlation between the feature map of the current frame and the feature map of the historical frame input to the enhanced short-term memory network, so as to better obtain the segmentation target of interest from the historical frame image. Valuable feature information.

In some embodiments, the feature pairs f _t input to the decoder are:

用于表征所述当前帧图像的特征对包括的当前键矩阵，

用于表征第i历史帧图像的特征对包括的第i历史键矩阵，

用于表征所述第i历史帧图像的特征对包括的第i历史值矩阵，

用于表征所述当前帧图像的特征对包括的值矩阵，

用于表征所述融合键矩阵。Wherein, R is the first correlation degree between the current key matrix and each historical key matrix, Z is the sum of all the first correlation degrees, and i represents the serial number of any historical frame image input into the enhanced short-term memory network , t represents the number of the current frame image and the total number of all the historical frame images and the current frame image,

The current key matrix included in the feature pair for characterizing the current frame image,

The i-th history key matrix included in the feature pair used to characterize the i-th historical frame image,

The i-th history value matrix included in the feature pair for characterizing the i-th historical frame image,

The value matrix included in the feature pair used to characterize the current frame image,

used to characterize the fusion bond matrix.

Some embodiments of the present application provide a calculation formula for quantitatively fusing feature pairs of historical frame images and feature pairs of current frame images.

In some embodiments, the first correlation is calculated by the following formula:

用于表征所述当前键矩阵和所述第i历史键矩阵的点乘操作。in,

A dot product operation for characterizing the current key matrix and the i-th historical key matrix.

Some embodiments of the present application use the e-exponential function to calculate the fusion result of the key matrix included in the feature pair of the current frame image and the key matrix included in the feature pair of each historical frame image.

In some embodiments, the total number of at least one historical frame image received by the enhanced short-term memory network is three.

Some embodiments of the present application implement inter-frame tracking by setting 3 frames of historical frame images, the amount of calculation is moderate, and the segmentation accuracy of the current frame image is effectively improved.

In some embodiments, the extracting features of at least one historical frame image adjacent to the current frame image to obtain feature pairs of each historical frame image in the at least one historical frame image, including: extracting the second historical frame image The image and the segmentation mask of the second historical frame image are concatenated and input into the enhanced short-term memory network to obtain feature pairs of the second historical frame image, wherein the second historical frame image is the Any image from at least one historical frame image.

Some embodiments of the present application achieve better inter-frame tracking by simultaneously inputting each frame of historical image and the segmentation result of each frame of historical image into an enhanced short-term memory network, and ultimately improve the segmentation result of the target of interest.

In a second aspect, some embodiments of the present application provide an apparatus for segmenting video objects, the apparatus comprising: a historical frame feature mining module configured to extract features of at least one historical frame image before the current frame image, and obtain the The feature pair of each historical frame image in the at least one historical frame image; the current frame encoding network module, configured to extract the feature of the current frame image, to obtain the feature pair of the current frame image; The decoding network module is configured to The feature pair of each historical frame image, the feature pair of the current frame image, and a decoder to obtain the segmentation mask image of the target of interest in the current frame image; wherein, the feature pair includes a key matrix and a value matrix .

In some embodiments, the current frame encoding network module is further configured to: extract features of the current frame image by using a convolution layer and a downsampling layer to obtain a current frame feature map; use the convolution layer and the The downsampling layer extracts the features of the cropped image, and obtains the cropped image feature map, wherein the cropped image is obtained by cropping the target of interest from the first frame image, and the first frame image is the first image in the video sequence. The frame of the target of interest appears; and the feature pair of the current frame image is obtained by fusing the current frame feature map and the cropped image feature map according to the feature similarity fusion module.

In a third aspect, some embodiments of the present application provide a network model training method for segmenting video objects. The network model training method for segmenting video objects includes: using a data set to perform salient object segmentation training on a basic network, wherein the The basic network includes a current frame encoder and a decoder; the video object segmentation network is trained according to at least one historical frame image, the current frame image and the cropped image, wherein the video object separate network includes the basic network and the enhanced network obtained after the training is completed. The current frame encoder is further configured to extract and fuse features of the current frame image and the cropped image, and the at least one historical frame image and the current frame image are from the same video sequence.

In a fourth aspect, some embodiments of the present application provide a system comprising one or more computers and one or more storage devices storing instructions, when the instructions are executed by the one or more computers, The one or more computers are caused to perform operations according to the respective methods described in the first aspect above.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, therefore It should not be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can also be obtained from these drawings without any creative effort.

1 is one of the architectural diagrams of a video object segmentation network provided by an embodiment of the present application;

FIG. 2 is the second architectural diagram of a video object segmentation network provided by an embodiment of the present application;

3 is one of the flowcharts of a method for segmenting a video object provided by an embodiment of the present application;

FIG. 4 is the second flowchart of the method for segmenting a video object provided by an embodiment of the present application;

FIG. 5 is the third architectural diagram of a video object segmentation network provided by an embodiment of the present application;

FIG. 6 is a block diagram of a composition of an apparatus for segmenting a video object provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

The related art video object segmentation methods are mainly divided into two categories. One class attempts to transfer the object segmentation results from the first frame to subsequent frames. For example, MaskTrack uses an optical flow method to simultaneously input the current frame image, the object mask mask of the current frame, and the image of the next frame to learn the change of the object mask mask of the next frame relative to the optical flow of the previous frame. . However, the accuracy of such methods will drop significantly when the object changes greatly, and the wrong segmentation results will seriously affect the segmentation accuracy of subsequent frames. Another type of method divides the video segmentation method into two parts: image segmentation and object tracking. First, instance segmentation is performed for each frame of image to distinguish each object, and then the object is identified by re-identification (re-identification) and other methods. The classification between them realizes the object correspondence of consecutive frames. Such methods are mainly affected by the accuracy of instance segmentation methods, and the results of instance segmentation are significantly worse than semantic segmentation on edges or small objects.

Different from the above-mentioned related art, the embodiments of the present application provide a more effective deep learning-based method for video object segmentation. For example, in some embodiments of the present application, based on an enhanced short-term memory network and a network that fuses the similarity between the current frame image feature and the cropped image feature, object segmentation and tracking are fused into a neural network, through The segmentation results of historical frames are used to supervise the segmentation results of current frames, which achieves a significant improvement in the accuracy of video object segmentation.

Please refer to FIG. 1. FIG. 1 is an architecture diagram of a video object segmentation network model based on a deep learning method provided by some embodiments of the present application. As shown in FIG. 1 , the architecture of the video object segmentation network model according to some embodiments of the present application includes an enhanced short-term memory network 100, a current frame encoder 120 and a decoder 130, wherein the decoder 130 directly or through other functional modules Connect with the enhanced short-term memory network 100 and the current encoder 120 .

The enhanced short-term memory network 100 includes one or more parallel historical frame encoders, wherein each historical frame encoder 110 is configured to receive an input historical frame image. As shown in FIG. 1, the enhanced short-term memory network 100 includes N historical frame encoders 110, wherein each historical frame encoder 110 is configured to receive an input historical frame image (eg, through the N historical frame encoders in FIG. 1 ). The historical frame encoder 110 respectively receives the inputted first historical frame image, the second historical frame image, ... up to the Nth historical frame image) and performs feature extraction on the inputted historical frame image to obtain the corresponding historical frame image. feature pair. Each historical frame encoder 110 includes a plurality of convolutional layers and a plurality of downsampling layers, and performs feature extraction on the input historical frame images through the convolutional layers and the downsampling layers to obtain a first feature pair, a second feature pair, . . . ... up to the Nth feature pair, where N is a natural number greater than 1 (eg, N takes the value 3 in some examples). It should be noted that the historical frame encoder 110 in FIG. 1 belongs to an encoder or an encoding network included in a semantic segmentation network. For example, each historical encoder 110 is an encoder included in the unet network (or referred to as "U-shaped convolutional neural network" or "U-shaped convolutional neural network", including an encoder encoder and a decoder decoder composed of convolutional layers). device part. The historical frame encoder 110 performs feature extraction on each historical frame image at different scales, and a first feature matrix and a second feature matrix corresponding to each historical frame image can be obtained, wherein the first feature matrix can be named as a historical frame key matrix , the second feature matrix can be named as the history frame value matrix.

The current frame encoder 120 in FIG. 1 is configured to receive an input current frame image, and extract features of the current frame image to obtain feature pairs of the current frame. For example, the current frame encoder 120 belongs to the encoder included in the semantic segmentation network. Specifically, the current frame encoder may be an encoder part included in the unet network. As an embodiment, the current frame encoder 120 includes: a plurality of convolutional layers and a plurality of downsampling layers, and extracting features of different scales on the current frame image through these layers can obtain a first feature matrix corresponding to the current frame image and a second feature matrix, where the first feature matrix may be named the current frame key matrix, and the second feature matrix may be named the current frame value matrix.

It should be noted that, in some embodiments of the present application, the enhanced short-term memory network 100 further includes a feature pair connection module 115 for connecting the feature pairs of corresponding historical frames extracted by each historical frame encoder. In other embodiments of the present application, the feature pair connection module 115 and the current frame encoder 120 are also connected to the historical feature fusion module 140 to realize the fusion of the feature pairs of the current frame image and the feature pairs of each historical frame to obtain fusion features The pair (ie the fusion key matrix below and the current value matrix included in the feature pair of the current frame image) are then input to the decoder 130 for decoding, and finally the segmentation mask image of the object of interest in the current frame image is obtained. For the fusion algorithm and process of the historical feature fusion module 140, please refer to the relevant description below, and to avoid repetition, no detailed description is given here.

In order to further highlight the target of interest for the current segmentation and improve the accuracy of video object segmentation, the architecture of the network model for video object segmentation according to some embodiments of the present application is shown in FIG. 2 .

Different from FIG. 1 , the current frame encoder 120 in FIG. 2 includes two parallel feature extraction units and a feature similarity fusion module 123 . In some embodiments of the present application, each of the two parallel feature extraction units includes a multi-layer convolution layer and a multi-layer downsampling layer (ie, the multi-layer convolution and downsampling network 121 of FIG. 2 ) , input the current frame image into the multi-layer convolution and down-sampling network 121 to obtain the current frame feature map, and at the same time input the cropped image containing the target of interest into the multi-layer convolution and down-sampling network 121 to obtain the cropped image feature map, and then pass The feature similarity fusion module 123 fuses the feature map of the current frame and the feature map of the cropped image to obtain a fusion feature, and obtains a feature pair of the current frame image based on the fusion feature (for example, extracting the fusion feature from a short-term memory network to obtain the feature of the current frame image. key-value pair).

It can be understood that, in order to perform video object segmentation by using the network model for video object segmentation in FIG. 1 , the network model in FIG. 1 needs to be trained first. In some embodiments of the present application, the method for training the video segmentation network model in FIG. 1 adopted by some embodiments of the present application includes: using a data set to perform salient object segmentation training on the basic network 20 in FIG. 1 , wherein the basic The network 20 includes a current frame encoder 120 and a decoder 130; the video object segmentation network 10 (ie, the entire network model in FIG. 1 ) is trained according to at least one historical frame image and the current frame image, wherein the video object segmentation network includes training The resulting base network and enhanced short-term memory network 100 upon completion.

In order to perform video object segmentation through the network model in FIG. 2 , the embodiment of the present application needs to first train the video object segmentation network model in FIG. 2 . In some embodiments of the present application, the method for training the network model in FIG. 2 according to some embodiments of the present application includes: using a data set to perform salient object segmentation training on the basic network 20 in FIG. 2 , wherein the basic network 20 includes Multi-layer convolution and downsampling network (implementing at least part of the function of the current frame encoder) and decoder 130; training the video object segmentation network 10 (ie the entire network in FIG. 2 ) according to at least one historical frame image, current frame image and cropped image model), wherein, the video object network 10 includes the basic network obtained after the training is completed, the enhanced short-term memory network and the model for obtaining the similarity of the current frame image and the cropped image feature (specifically including FIG. 2 for The multi-layer convolution and downsampling network 121 and the feature similarity fusion module 123 of FIG. 2 ) receiving the input cropped image), the current frame encoder 120 is further configured to extract the current frame image and the features of the cropped image and fusion.

That is to say, based on the network architecture of FIG. 1 or FIG. 2 , the embodiments of the present application are trained and tested on public datasets. Since there are many types of objects to be segmented and are not known, some embodiments of the present application first train a saliency object segmentation using the basic network model. The purpose of salient object segmentation is to segment the salient objects in the image, regardless of the type of objects. For example, the embodiment of the present application adopts COCO, VOC, and other salient data sets, and randomly selects an object from the labeled images as a salient object, and removes other objects. Based on the generated data set, a basic network for single object segmentation is trained, and then the basic network is used as initialization to train the video object segmentation network 10 in FIG. 1 or FIG. 2 . In some embodiments of the present application, the segmentation accuracy evaluation results obtained by the video object segmentation network 10 tested on the youtube vos data set are better than other evaluation methods.

It should be noted that at least one historical frame image and the current frame image involved in training the video object segmentation network model of the embodiment of the present application are from the same video sequence, but each historical frame image in the at least one historical frame image is random. Selected from the video sequence, these historical frame images and the current frame image are not necessarily adjacent images.

The process of the video object segmentation method performed by the trained video object segmentation network 10 is exemplarily explained below with reference to FIG. 3 .

As shown in FIG. 3 , some embodiments of the present application provide a method for segmenting a video object. The method includes: S101 , extracting features of at least one historical frame image before the current frame image, and obtaining the at least one historical frame image The feature pair of each historical frame image in the frame image; S102, extract the feature of the current frame image to obtain the feature pair of the current frame image; S103, according to the feature pair of each historical frame image, the feature pair of the current frame image and The decoder obtains the segmentation mask of the target of interest in the current frame image; wherein, each historical frame image in the at least one historical frame image is the previous frame or multiple frame images of the current frame image, so The feature pair includes a key matrix and a value matrix. Some embodiments of the present application implement inter-frame tracking through an enhanced short-term memory network, which significantly improves the video object segmentation accuracy for the current frame image. In some embodiments of the present application, S101 extracts the feature of at least one historical frame image according to the enhanced short-term memory network, or S102 uses the current frame encoder to extract the feature of the current frame image.

It should be noted that at least one historical frame image input to the enhanced short-term memory network and the current frame image input to the current frame encoder are from the same video sequence, and each historical frame image included in the at least one historical frame image An image is the frame or frames preceding the current frame image in the video sequence. That is to say, in some embodiments of the present application, when using the trained video object segmentation network to perform video object segmentation on the current frame image, the input of at least one historical frame image of the enhanced short-term memory network and the input current frame encoder The current frame image of is several adjacent frame images. In other embodiments of the present application, when using the trained video object segmentation network to perform video object segmentation on the current frame image, at least one historical frame image input to the enhanced short-term memory network and the current frame input to the current frame encoder are used. The images are non-adjacent images (for example, a historical frame image is extracted every n frames before the current frame image). When the number of historical frames that need to be input to the enhanced short-term memory network is greater than the total number of frames before the current frame image, at least one historical frame may include two identical historical frames. For example, if the number of input nodes of the enhanced short-term memory network is 3, the first 3 frames of the current frame image can be taken when segmenting the video object of the current frame image. If there are less than 3 frames, the first frame or 2 frames can be repeatedly selected .

The steps of FIG. 3 are exemplified below.

In some embodiments of the present application, S101 includes: concatenating any historical frame image (eg, the second historical frame image) and the pre-acquired segmentation mask of the historical frame image into the enhanced short-term memory network, including: A historical frame encoder of , to obtain the feature pair of the historical frame image. For example, when the second historical frame image is an RGB three-channel image, the information input to each historical frame encoder in the enhanced short-term memory network is the information including R channel, G channel, B channel and the segmentation mask of the corresponding historical frame image. Four-channel image, and the output of the historical frame encoder is two feature maps, and the matrices corresponding to these two feature maps can be named as key matrix and value matrix.

For example, the enhanced short-term memory network of S101 includes at least one encoder of the semantic segmentation network (ie, the historical frame encoder of FIG. 1 or FIG. 2 ), the encoders in the at least one encoder are connected in parallel, and each encoder The device receives an input historical frame image and the segmentation result of the historical frame image. After that, each encoder extracts the features of the input historical frame images, respectively obtains the feature pair results of each historical frame image and outputs them.

That is to say, as mentioned above, the enhanced short-term memory network structure adopted in S101 can be obtained by paralleling encoders included in semantic segmentation networks such as unet, and each encoder receives an input historical frame image to obtain The feature map corresponding to this frame of image, in some embodiments of the present application, the feature map includes two feature matrices, which are a key matrix and a value matrix, respectively. For example, an encoder includes multiple convolutional and downsampling layers.

In some embodiments of the present application, as shown in FIG. 4 , the current frame encoder in S102 includes a convolution layer, a downsampling layer, and a feature similarity fusion module; wherein, S102 includes: S1021 , using the convolution layer Extracting features of the current frame image with the downsampling layer to obtain a feature map of the current frame; S1022, using the convolutional layer and the downsampling layer to extract features of the cropped image to obtain a cropped image feature map, wherein the The cropped image is obtained by cropping the target of interest from the first frame of image, and the first frame of image is the frame in which the target of interest appears for the first time in the video sequence; S1023, according to the feature similarity fusion module, the The current frame feature map and the cropped image feature map are fused to obtain a fused image, and a feature pair of the current frame image is obtained based on the fused image.

It should be noted that the process of obtaining the feature map of the current frame and the feature map of the cropped image can be performed simultaneously, or the feature map of the cropped image can be obtained first and then the feature map of the current frame. The embodiment of the present application does not limit S1021 to obtain the feature map of the current frame and S1022 acquires the sequence of the two steps of cropping the image feature map. Therefore, in some embodiments, S102 includes: using the convolution layer and the down-sampling layer to extract features of the cropped image to obtain a cropped image feature map, wherein the cropped image is cropped from the first frame of image obtained from the target of interest, the first frame image is the frame in which the target of interest appears for the first time in the video sequence; the convolution layer and the downsampling layer are used to extract the features of the current frame image, and the current frame image is obtained. frame feature map; according to the feature similarity fusion module, the current frame feature map and the cropped image feature map are fused to obtain a fused image, and a feature pair of the current frame image is obtained based on the fused image.

In order to further highlight the segmented target of interest, the feature similarity fusion module in S102 fuses the current frame feature map and the cropped image feature map through a short-term memory network to obtain the fused image, and based on the fused image Get the feature pair of the current frame image. For example, the feature similarity fusion module obtains the fusion image of S1023 according to the following formula:

表征融合图像对应的矩阵，feat_t表征与当前帧特征图对应的矩阵，feat_p表征与裁剪图像特征图对应的矩阵。in,

represents the matrix corresponding to the fusion image, feat _t represents the matrix corresponding to the feature map of the current frame, and feat _p represents the matrix corresponding to the feature map of the cropped image.

In some embodiments of the present application, S103 includes: respectively performing a fusion operation on the current key matrix included in the feature pair of the current frame image and the historical key matrix included in the feature pair of each historical frame image to obtain a fusion key matrix ; Inputting the fusion key matrix and the current value matrix (ie the feature pair input to the decoder) included in the feature pair of the current frame into the decoder to obtain the segmentation mask image. For example, performing a fusion operation on the current key matrix included in the feature pair of the current frame and the historical key matrix included in the feature pair of each historical frame image respectively, to obtain a fusion key matrix, includes: obtaining the current key matrix and The first correlation degree of each historical key matrix in all the historical key matrices; the fusion key matrix is obtained according to the first correlation degree. That is, the fusion key matrix and the current value matrix included in the feature pair of the current frame are taken as the feature pair of the input decoder.

Specifically, the feature pair ft input to the _decoder is:

用于表征当前帧图像的特征对包括的当前键矩阵，

用于表征第i历史帧图像的特征对包括的第i历史键矩阵，

用于表征第i历史帧图像的特征对包括的第i历史值矩阵，

用于表征当前帧图像的特征对包括的值矩阵。需要说明的是，第i历史帧图像中的编号i并不是这帧图像在整个视频序列中的帧序号，而是根据每次输入增强型短时记忆网络的总张数进行的编号，

用于表征所述融合键矩阵。Among them, R is the first correlation degree between the current key matrix and each historical key matrix, Z is the sum of all the first correlation degrees, i represents the number of any historical frame image input into the enhanced short-term memory network, t Characterizing the number of the current frame image and the total number of all the historical frame images and the current frame image,

The current key matrix including the feature pair used to characterize the current frame image,

The i-th history key matrix included in the feature pair used to characterize the i-th historical frame image,

The i-th history value matrix including the feature pair used to characterize the i-th historical frame image,

A matrix of values comprised of feature pairs used to characterize the current frame image. It should be noted that the number i in the i-th historical frame image is not the frame number of this frame image in the entire video sequence, but is numbered according to the total number of sheets input to the enhanced short-term memory network each time,

used to characterize the fusion bond matrix.

For example, the first correlation degree is calculated by the following formula:

用于表征当前键矩阵和第i历史键矩阵的点乘操作。本申请的一些实施例采用e指数函数计算当前帧图像的特征对包括的键矩阵和各历史帧图像的特征对包括的键矩阵的融合结果。需要说明的是，还可以采用其它的函数类型计算第一相关度，例如，使用矩阵点乘后的平方根或绝对值等计算相似度。in,

A dot product operation used to characterize the current key matrix and the i-th historical key matrix. Some embodiments of the present application use the e-exponential function to calculate the fusion result of the key matrix included in the feature pair of the current frame image and the key matrix included in the feature pair of each historical frame image. It should be noted that other function types may also be used to calculate the first correlation, for example, the similarity is calculated by using the square root or absolute value of matrix dot product.

Some embodiments of the present application provide a method for fusing the features of the mined historical frame image and the features of the mined current frame image, and the fused feature pair (that is, the fusion key matrix and the current frame feature) obtained by this fusion method is Decoding the included current value matrix) can improve the accuracy of the segmentation mask decoded with the decoder.

Taking three historical frame images as an example, the video object segmentation method and the corresponding video segmentation network architecture according to some embodiments of the present application are exemplarily described with reference to FIG. 5 .

As shown in FIG. 5 , the video object segmentation network architecture for implementing the video object segmentation of the present application includes a backbone network, that is, a grid formed by the current frame encoder 120 and the decoder 130 in FIG. 5 . As shown in the right part of Figure 5, the backbone network acts like a standard semantic segmentation network. There are many kinds of semantic segmentation networks. In the video segmentation network provided by the embodiments of the present application, the backbone network may be any semantic segmentation network structure. For example, some embodiments of the present application employ a standard unet structure. The input to the unet network is an image. Through the multi-layer convolution and down-sampling feature network (ie the current frame encoder 120 in Figure 5), the features of different scales of the image are extracted, and then through a continuous up-sampling decoding network (ie Figure 5). 5), which fuses features of different scales and outputs an object segmentation mask (that is, the final output of the entire network model in Figure 5).

As shown in FIG. 5 , the enhanced short-term memory network 100 in this embodiment of the present application is to implement tracking between frames. For the current frame image It, its historical frame and historical segmentation results are [I0,...,It-1] and [M0,...,Mt-1], respectively. For each frame of image, the enhanced short-term memory network 100 will extract a feature pair of [k, v] (that is, the encoder Enc_m (Memory encoder) of the encoder memory network in FIG. 5 extracts features from each historical frame image to obtain The key matrix (that is, the Key matrix of Fig. 5) and the value matrix (that is, the Value matrix of Fig. 5). For example, each frame image included in the video sequence is an RGB image, then each historical frame included in the enhanced short-term memory network 100 The input of the encoder (Encm) is the 4-channel image after I and M are concatenated (that is, including R, G, B and the segmentation result mask image of the frame), and the output is two feature maps or two features. matrix, namely the key matrix Key and the value matrix Value. The video object segmentation network in Figure 5 also includes a feature pair connection module Concat.

It should be noted that, in order to ensure that the input and output of the video object segmentation network are consistent in actual use, during the training process, 3 frames of images are randomly selected from the historical frames and input to the enhanced short-term memory network 100, and the video object segmentation network is carried out. train. During implementation, the first three frames of the current frame image may be selected, and if there are less than three frames, the previous historical frame image or the previous two historical frame images will be repeatedly selected. For the current frame image, while extracting features of different scales, the backbone network also generates a [k, v] feature pair, that is, the key matrix and value matrix in Figure 5.

When performing video object segmentation in some embodiments of the present application, it is necessary to fuse the feature pair [k, v] of the extracted three historical frame images and the extracted feature pair [k, v] of the current frame image to obtain a new feature pair [k, v]. The fusion feature map featmap of , takes the fusion feature map featmap as the input of the decoder 130 (or called decoding network) in the subsequent backbone network. For example, the specific fusion method of the feature pair of the historical frame image and the feature pair of the current frame is represented by a formula, where t is the serial number of the current frame image, i is the serial number of the historical frame image input to the enhanced short-term memory network, t =3, i takes the values 0, 1 and 2:

Among them, R is the correlation of the two featmaps, and Z is the sum of the correlations of all frames.

It should be noted that, for the definitions of the relevant parameters involved in the above formulas, reference may be made to the foregoing descriptions, and to avoid repetitions, detailed descriptions are omitted here.

Further, in order to better highlight the currently segmented object, some embodiments of the present application also design a feature similarity module (ie, the feature similarity fusion module 123 in FIG. 5 ) that fuses the current frame image feature and the cropped image feature. The current frame encoder 120 in FIG. 5 includes: a current frame encoder 121 composed of multi-layer convolutional layers and downsampling layers, a feature fusion module 123 and a short-term memory network 122, wherein the current frame encoder 121 is used to obtain the current frame encoder 121. The feature of the frame image and the feature of the cropped image, the feature fusion module 123 is used to fuse the current frame feature map and the cropped image feature map processed by the current frame encoder 121, and the features fused by the feature fusion module 123 use a short-term memory network. 122 to get a key-value pair consisting of matrix k and matrix v. For example, input the cropped image containing the target of interest into the multi-layer convolutional layer and the downsampling layer (ie the current frame encoder 121) to obtain a cropped image feature map featmap, input the current frame image to the multi-layer convolutional layer and The current frame feature map is obtained in the downsampling layer (that is, the current frame encoder 121 included in the feature extraction network), and then the current frame feature map featmap extracted by the cropped image feature map featmap and the backbone network adopts the feature fusion module 123 according to the following: The fusion formula is fused, and the new featmap obtained after fusion is entered into the short-term memory network 122 to obtain the key-value pair key and Value.

In some embodiments of the present application, the method for the feature fusion module 123 to fuse the current frame feature map and the cropped image feature map is to multiply the two feature maps featmaps of the current frame feature map and the cropped image feature map, and connect them in series. The original backbone feature (that is, the feature map of the current frame is connected in series), the specific fusion formula is as follows, wherein, the meaning of each parameter in the following fusion formula can be referred to the above description to avoid repetition.

It should be noted that, in order to obtain a cropped image, some embodiments of the present application also need to put the first frame of image (that is, the frame in which the target of interest appears for the first time in the same video sequence, rather than the absolute first frame of image in the video sequence). The target object of interest is cropped out, and the place other than the object mask is set to 0, only the texture information of the object is retained, and the cropped image of the input multi-layer convolution and downsampling network 121 as shown in Figure 5 is obtained. The reading module 160 of FIG. 5 is configured to read historical key-value pairs of historical frame images, and read key-value pairs of the current frame fused with the current frame image and the cropped image, ASSP (ie atrous spatial pyramid pooling Pooling) is configured to fuse the historical key-value pair with the key-value pair of the current frame to obtain further features.

In some embodiments of the present application, the current frame encoder 120 may employ a Siamese query encoder, where the twin refers to the sharing of parameters of the two encoders.

Please refer to FIG. 6. FIG. 6 shows an apparatus for segmenting a video object provided by an embodiment of the present application. It should be understood that the apparatus corresponds to the method embodiment of FIG. 3 and can perform various steps involved in the above method embodiment. For the specific functions of the device, reference may be made to the above description, and to avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory or solidified in the operating system of the device in the form of software or firmware. The device for segmenting video objects includes: a historical frame feature mining module 101, configured to extract the current frame The feature of at least one historical frame image before the image is obtained, and the feature pair of each historical frame image in the at least one historical frame image is obtained; the current frame encoding network module 102 is configured to extract the feature of the current frame image to obtain The feature pair of the current frame image; the decoding network module 103 is configured to obtain the segmentation of the object of interest in the current frame image according to the feature pair of each historical frame image, the feature pair of the current frame image and the decoder A mask image; wherein each historical frame image in the at least one historical frame image is the previous frame or images of the current frame image, and the feature pair includes a key matrix and a value matrix.

In some embodiments of the present application, the current frame encoding network module 102 is further configured to: extract features of the current frame image by using a convolution layer and a downsampling layer to obtain a feature map of the current frame; layer and the downsampling layer to extract the features of the cropped image to obtain a cropped image feature map, wherein the cropped image is obtained by cropping the target of interest from the first frame of image, and the first frame of image is the The frame of the target of interest appears for the first time in the video sequence; the feature pair of the current frame is obtained by fusing the feature map of the current frame and the feature map of the cropped image according to the feature similarity fusion module.

Some embodiments of the present application provide a network model training method for segmenting video objects, the method for training a network model for video object segmentation includes: using a data set to perform salient object segmentation training on a basic network, wherein the basic network includes a current Frame encoder and decoder; video object segmentation network is trained according to at least one historical frame image, current frame image and cropped image, wherein the video object separate network includes a basic network and an enhanced short-term memory network obtained after training is completed , the current frame encoder is further configured to extract and fuse the features of the current frame image and the cropped image, and the at least one historical frame image and the current frame image are from the same video sequence.

Some embodiments of the present application provide a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to The computer or computers perform operations according to the respective methods described above with respect to FIG. 3 .

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architectures, functions and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present application. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above descriptions are merely examples of the present application, and are not intended to limit the protection scope of the present application. For those skilled in the art, various modifications and changes may be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application. It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Claims (14)

1. A method of segmenting video objects, the method comprising:

extracting the characteristics of at least one historical frame image before the current frame image to obtain the characteristic pair of each historical frame image in the at least one historical frame image;

extracting the characteristics of the current frame image to obtain a characteristic pair of the current frame image;

acquiring a segmentation mask of an interest target in the current frame image according to the feature pair of each historical frame image, the feature pair of the current frame image and a decoder;

wherein the feature pair comprises a key matrix and a value matrix.

2. The method of claim 1, wherein the features of the current frame image are extracted by a current frame encoder, the current frame encoder comprising a convolutional layer, a downsampling layer, and a feature similarity fusion module; wherein,

the extracting the features of the current frame image by using the current frame encoder to obtain the feature pairs of the current frame image comprises the following steps:

extracting the characteristics of the current frame image by adopting the convolution layer and the down-sampling layer to obtain a current frame characteristic diagram;

extracting features of a cut image by adopting the convolution layer and the downsampling layer to obtain a cut image feature map, wherein the cut image is obtained by cutting the interested target from a first frame image, and the first frame image is a frame of the video sequence in which the interested target appears for the first time;

and fusing the current frame feature map and the cut image feature map according to the feature similarity fusion module to obtain a fused image, and obtaining a feature pair of the current frame image based on the fused image.

3. The method of claim 2, wherein the feature similarity fusion module fuses the current frame feature map and the cropped image feature map through a short-time memory network to obtain the fused image.

4. The method of claim 2 or 3, wherein the feature similarity fusion module obtains the fused image according to the following formula:

wherein,

characterizing the matrix, feat, to which the fused image corresponds_tCharacterizing a matrix, feat, corresponding to the current frame profile_pAnd characterizing a matrix corresponding to the feature map of the cut image.

5. The method according to any one of claims 1-4, wherein the features of the current frame image are extracted through an enhanced short-time memory network, the enhanced short-time memory network comprises at least one encoder of a semantic segmentation network, the encoders of the at least one encoder are connected in parallel, and each encoder receives an input historical frame image and a segmentation result of the historical frame image.

6. The method according to any one of claims 1 to 4, wherein the obtaining a segmentation mask of the object of interest in the current frame image according to the feature pair of the history frame images, the feature pair of the current frame image and a decoder comprises:

respectively carrying out fusion operation on the current key matrix included by the feature pairs of the current frame image and the historical key matrix included by the feature pairs of each historical frame image to obtain a fusion key matrix;

and inputting the current value matrixes included by the fusion key matrix and the feature pairs of the current frame image into the decoder to obtain the segmentation mask.

7. The method of claim 6, wherein the performing a fusion operation on the current key matrix included in the feature pair of the current frame image and the historical key matrix included in the feature pair of each historical frame image to obtain a fused key matrix comprises:

respectively acquiring first relevancy of the current key matrix and each historical key matrix in all the historical key matrices;

and obtaining the fusion key matrix according to the first correlation.

8. The method of claim 6, wherein the pair of features f input to the decoder_tComprises the following steps:

wherein R is the first correlation degree of the current key matrix and each historical key matrix, Z is the sum of all the first correlation degrees, i represents the number of any one historical frame image input into the enhanced short-time memory network, t represents the number of the current frame image and the total number of all the historical frame images and the current frame image,

a current key matrix for characterizing feature pairs of the current frame image,

an ith history key matrix included in the feature pairs for characterizing the ith history frame image,

for watchesAn ith history value matrix included in a feature pair characterizing the ith history frame image,

a matrix of values comprised by pairs of features characterizing the current frame image,

for characterizing the fusion key matrix.

9. The method of claim 8, wherein the first degree of correlation is calculated by the formula:

wherein,

a dot product operation for characterizing the current key matrix and the ith historical key matrix.

10. The method of claim 1, wherein a total number of the at least one historical frame image is 3.

11. The method according to any one of claims 1 to 10, wherein the extracting features of at least one historical frame image adjacent to the current frame image to obtain a feature pair of each historical frame image in the at least one historical frame image comprises: and connecting the segmentation masks of a second historical frame image and the second historical frame image in series and inputting the second historical frame image and the segmentation masks of the second historical frame image into an enhanced short-time memory network to obtain a feature pair of the second historical frame image, wherein the second historical frame image is any one of the at least one historical frame image.

12. An apparatus for segmenting video objects, the apparatus comprising:

the historical frame feature mining module is configured to extract features of at least one historical frame image before a current frame image to obtain feature pairs of each historical frame image in the at least one historical frame image;

the current frame coding network module is configured to extract the features of the current frame image to obtain a feature pair of the current frame image;

the decoding network module is configured to acquire a segmentation mask image of an interested target in the current frame image according to the feature pairs of the historical frame images, the feature pairs of the current frame image and a decoder;

wherein the feature pair comprises a key matrix and a value matrix.

13. A network model training method for segmenting video objects is characterized by comprising the following steps:

carrying out salient object segmentation training on a basic network by adopting a data set, wherein the basic network comprises a current frame encoder and a decoder;

training a video object segmentation network according to at least one historical frame image, a current frame image and a cut image, wherein the video object networks respectively comprise a basic network and an enhanced short-time memory network which are obtained after training is completed, the current frame encoder is further configured to extract and fuse the characteristics of the current frame image and the cut image, and the at least one historical frame image and the current frame image are from the same video sequence.

14. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective methods of any of claims 1-11.

Images