CN109461167B - Image processing model training method, mapping method, device, medium and terminal - Google Patents

Abstract

The embodiment of the application discloses a training method, a matting method, a device, a medium and a terminal of an image processing model. The training method of the image processing model comprises the steps of obtaining a trimap image of an original image; generating a training sample set according to the original image and the trimap image; and training a preset deep learning network based on the training sample set to obtain an image processing model, wherein the image processing model is used for performing labeling processing on an original image to obtain a trisection image. By adopting the technical scheme, the input original image can be automatically labeled to obtain the three-part graph, a large amount of hair-level data labeling is not required to be carried out in a manual labeling mode, the labeling workload can be reduced, and the image labeling efficiency is improved. In addition, the original image is marked by adopting the image processing model, so that errors possibly introduced by manual marking are avoided, and the matting effect can be optimized.

Description

Image processing model training method, mapping method, device, medium and terminal

technical field

The embodiments of the present application relate to image processing technologies, and in particular, to a training method, a matting method, an apparatus, a medium, and a terminal for an image processing model.

Background technique

At present, matting has become one of the most common operations in image processing. For example, more and more people choose to buy clothes on the Internet, so the e-commerce search function came into being. It is very difficult to accurately search for similar clothing, so it is necessary to segment the portraits in the pictures. For another example, the realization of some portrait beautification functions also depends on the precise segmentation of the background and the portrait.

The matting scheme in the related art is mainly based on the pixel clustering method and the algorithm based on the "Graph Partitioning" (Graph Partitioning) algorithm. This kind of method is more complex in the background or the background is similar to the foreground (also called the matting target). In the case of a large degree, the segmentation effect is not good, such as the perfect segmentation of hair, delicate clothes, branches and other delicate objects cannot be achieved.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a training method, a matting method, an apparatus, a medium, and a terminal for an image processing model, which can optimize the matting scheme in the related art.

In a first aspect, an embodiment of the present application provides a training method for an image processing model, including:

Get a tripartite map of the original image;

generating a training sample set according to the original image and the tripartite map;

A preset deep learning network is trained based on the training sample set to obtain an image processing model, wherein the image processing model is used for labeling the original image to obtain a tripartite map.

In the second aspect, the embodiments of the present application also provide a method for matting, including:

Obtain the target image to be cutout;

An image processing model is used to label the target image to obtain a tripartite map of the target image, wherein the image processing model is a deep learning network trained by a training sample set composed of the original image and the tripartite map;

Based on the target picture and the three-part map, a set cutout algorithm is used to perform cutout processing on the target picture to obtain a cutout image.

In a third aspect, the embodiments of the present application also provide an apparatus for training an image processing model, including:

Tripartite image acquisition module, used to obtain the tripartite map of the original image;

a sample generation module, configured to generate a training sample set according to the original image and the tripartite map;

The model training module is used for training a preset deep learning network based on the training sample set to obtain an image processing model, wherein the image processing model is used for labeling the original image to obtain a tripartite map.

In a fourth aspect, an embodiment of the present application also provides a drawing device, comprising:

The target image acquisition module is used to acquire the target image to be cut out;

The image labeling module is configured to perform labeling processing on the target image through an image processing model to obtain a tripartite image of the target image, wherein the image processing model is trained on a training sample set composed of the original image and the tripartite image. deep learning network;

The matting module is configured to perform matting processing on the target image based on the target image and the three-part image by using a preset matting algorithm to obtain a matting image.

In a fifth aspect, the embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored; when the program is executed by the processor, the training method of the image processing model described in the embodiments of the present application is implemented, or , and when the program is executed by the processor, the image matting method described in the embodiments of the present application is implemented.

In a sixth aspect, an embodiment of the present application provides a terminal, including a memory, a processor, and a computer program stored in the memory and running on the processor; when the processor executes the computer program, the implementation is as described in the embodiments of the present application The training method of the image processing model, or, when the processor executes the computer program, implements the image matting method described in the embodiments of the present application.

The embodiment of the present application provides a training scheme for an image processing model, obtaining a tripartite map of an original image, and generating a training sample set according to a plurality of original images and corresponding tripartite maps; based on the training sample set, a preset deep learning network Perform training to iteratively update the parameter values of the deep learning network. After the training is completed, an image processing model is obtained, and the original image can be labeled and processed through the image processing model to obtain a tripartite map. By adopting the above technical solution, the deep learning network can be trained based on the original image and the corresponding tripartite graph, so that it can automatically label the input original image to obtain the tripartite graph, without manually labeling a large number of hairs. Level data annotation can reduce the workload of annotation and improve the efficiency of image annotation. In addition, the image processing model is used to annotate the original image to avoid errors that may be introduced by manual annotation, which can improve the effect of optimizing the matting.

Description of drawings

1 is a flowchart of a training method for an image processing model provided by an embodiment of the present application;

2 is a flowchart of another method for training an image processing model provided by an embodiment of the present application;

3 is a flow chart of a method for matting according to an embodiment of the present application;

4 is a schematic structural diagram of an apparatus for training an image processing model according to an embodiment of the present application;

5 is a structural block diagram of a matting device provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a terminal according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of another terminal provided by an embodiment of the present application;

FIG. 8 is a structural block diagram of a smart phone according to an embodiment of the present application.

Detailed ways

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all the structures related to the present application.

Before discussing the exemplary embodiments in greater detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowchart depicts the steps as a sequential process, many of the steps may be performed in parallel, concurrently, or concurrently. Furthermore, the order of the steps can be rearranged. The process may be terminated when its operation is complete, but may also have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

FIG. 1 is a flowchart of an image processing model training method provided by an embodiment of the present application. The method can be performed by a training device for an image processing model, wherein the device can be implemented by software and/or hardware, and can generally be integrated in terminal or server. As shown in Figure 1, the method includes:

Step 110: Obtain a tripartite map of the original image.

It should be noted that the terminals in the embodiments of the present application may include smart devices such as mobile phones, tablet computers, notebook computers, and desktop computers.

It should be noted that a tripartite map, also known as a trimap map, is a contour image that annotates the edges of the target object in the image. The contour of the target object in the original image is manually annotated based on the set cutout algorithm and user input operation, thereby providing constraint information for the cutout operation. For example, the original image is roughly divided by the TRIMAP algorithm to divide the original image into foreground, background and unknown area to be determined, and white represents the foreground, black represents the background, and gray represents the unknown area to be determined, and the trimap image is obtained. .

Exemplarily, label information for pixels in the original image is acquired, and a tripartite map of the original image is generated according to the label information. Wherein, the annotation information may be the outline or boundary between the foreground and the background in the original image that is manually annotated.

Step 120: Generate a training sample set according to the original image and the tripartite map.

Exemplarily, the original image is associated with the corresponding trimap image as a training sample, and a training sample set is formed by a set number of training samples. Wherein, the set quantity can be the default quantity of the system.

Since the trimap map is obtained by manually labeling the set area in the original image, there may be labeling errors that are not easily noticed by the human eye during manual labeling, resulting in poor effect of the trimap operation based on the trimap map. In view of this problem, we can first verify whether the trimap map meets the set conditions, and judge whether the trimap map and the corresponding original image are used as training samples according to the verification result. The setting condition may be that the score of the cutout image exceeds the set score threshold, etc., which is not specifically limited in this embodiment of the present application. Assuming that the trimap image satisfies the set conditions, the trimap image and the corresponding original image are used as training samples, and the training sample set is composed of a set number of training samples.

Exemplarily, based on the original image and the trimap image, a preset matting algorithm is used to perform matting processing on the original image to obtain a matting image. Obtain the evaluation information of the cutout image, and judge whether the trimap image satisfies the set condition according to the evaluation information. In this embodiment of the present application, the evaluation information may be data for evaluating the matting effect of the matting image, such as scoring the matting image, or sorting the matting images according to the matting effect, and so on.

There are many ways to obtain evaluation information, which are not specifically limited in this embodiment of the present application. In some examples, the evaluation information may be obtained by analyzing user operations. For example, after the matting image is generated, if a user's correction operation on the matting image is detected, a lower score (a score lower than a set threshold) is given to the matting image. The cutout image may also be scored according to the number of positions of the user's correction operation, that is, the more positions the user corrects, the lower the score of the cutout image. The score of the cutout image is compared with the set score threshold, and when the score exceeds the set score threshold, it is determined that the trimap image satisfies the set condition.

Step 130: Train a preset deep learning network based on the training sample set to obtain an image processing model.

In this embodiment of the present application, the image processing model can automatically perform annotation processing on the input original image to obtain a tripartite map.

It should be noted that deep learning refers to a set of algorithms that use various machine learning algorithms on a multi-layer neural network to solve various problems in documents such as images and texts. Deep learning can be classified into neural networks in general, but there are many changes in the specific implementation. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through hierarchical networks, so as to solve the important problems that required manual design of features in the past. Deep learning is a framework that includes several important algorithms, such as Convolutional Neural Networks (CNN), AutoEncoder, Sparse Coding, Restricted BoltzmannMachine , referred to as RBM), deep belief network (Deep Belief Networks, referred to as DBN), and multilayer feedback recurrent neural network (Recurrent neural Network, referred to as RNN). For different problems (such as images, speech, and text, etc.), different frameworks can be selected, or frameworks can be selected based on factors such as computing speed and computing accuracy.

The parameters of the deep learning network include the weights of the edges between the layers in the hierarchical network and the bias θ of neurons, etc.

Exemplarily, the preset deep learning network in this application may be a deep learning model in which a post-processing layer is added after the output layer of the deep learning network based on image semantic segmentation. It should be noted that the image is composed of many pixels, and the meaning of semantic segmentation is to group/segment the pixels according to the different semantic meanings expressed in the image. The deep convolutional neural network model can be trained by using the segmented sample images to obtain a deep learning network based on image semantic segmentation, so as to realize the segmentation operation based on image semantics on the original image through the deep learning network based on semantic segmentation.

Taking the original image containing the portrait as an example, taking the portrait as the target image, in order to achieve the purpose of segmenting the portrait and the background, a deep convolutional neural network model can be used to perform semantic segmentation on the original image, and the obtained output image is a portrait with white pixels. represents the category map with the background represented by black pixels.

The post-processing layer is a layer after the output layer of the deep learning network based on image semantic segmentation. Segmentation thresholding results in a tripartite map with three colors of black, white, and gray. The segmentation thresholding process may be an operation of dividing the pixels included in the category map into several categories based on the threshold segmentation method. For example, the pixel points of the category map with black and white colors can be divided according to the gray level to obtain 3 pixel sets, the pixel points in each pixel set belong to the same threshold range interval, and the pixels of different pixel sets Pixels belong to different threshold ranges.

Exemplarily, based on the training sample set, a forward propagation algorithm is used to train the above-mentioned preset deep learning network (that is, inputting the original image into the preset deep learning network) to obtain an actual output result. Based on the back-propagation algorithm, the difference between the actual output result and the ideal output result (ie, the tripartite graph in the training sample set) is calculated. The parameter values of the deep learning network are adjusted by the back-propagation algorithm according to the method of minimizing the error.

It should be noted that, if the above-mentioned training method for the image processing model is executed by the server, after the training is completed, the server sends the image processing model to the terminal. When the model update event is triggered, the server updates the image processing model and sends the updated image processing model to the terminal. Wherein, the server receives the image processing model fed back by the terminal to obtain the annotation effect information of the trimap image by performing annotation on the original image. If the number of feedback messages with poor labeling effect exceeds the set threshold, the update operation of the image processing model is started.

Optionally, if the computing capability of the terminal is strong enough, the above training method of the image processing model may also be executed by the terminal. Since the user's evaluation of the cutout effect is subjective, some users think that the cutout effect is better, while others may not agree. By executing the training method of the image processing model on the terminal, the image processing model can be updated in time according to the personalized needs of the user, so as to obtain a model suitable for the terminal user.

In the technical solution of the embodiment of the present application, a tripartite map of an original image is obtained, and a training sample set is generated according to a plurality of original images and corresponding tripartite maps; based on the training sample set, a preset deep learning network is trained to iteratively update The parameter values of the deep learning network are obtained after training to obtain an image processing model, which can be used to label the original image to obtain a tripartite map. By adopting the above technical solution, the deep learning network can be trained based on the original image and the corresponding tripartite graph, so that it can automatically label the input original image to obtain the tripartite graph, without manually labeling a large number of hairs. Level data annotation can reduce the workload of annotation and improve the efficiency of image annotation. In addition, the image processing model is used to annotate the original image to avoid errors that may be introduced by manual annotation, which can improve the annotation effect.

FIG. 2 is a flowchart of another image processing model training method provided by an embodiment of the present application. As shown in FIG. 2 , the method includes:

Step 201: Obtain a tripartite map of the original image.

Exemplarily, a related operation performed by the user on the original image in order to separate the foreground and the background is obtained, and a trimap map of the original image is generated according to the operation.

Step 202 , based on the original image and the three-part map, use a preset matting algorithm to perform matting processing on the original image to obtain a matting image.

It should be noted that matting is usually an ultra-fine image segmentation technique for segmenting the foreground from the background (generalized image segmentation may also include separation between objects of equal status, etc.). The matting algorithm is an algorithm applied to the above image segmentation technology, including but not limited to Bayesian matting algorithm, Poisson matting algorithm, Grabcut segmentation algorithm, and domain information-based matting algorithm (Shared Sampling for Real Time). Alpha Matting), robust matting algorithm (robust matting) and lazysnapping algorithm. Different matting algorithms can be selected according to different original images, or an appropriate matting algorithm can be selected in combination with the original image, the matting effect and the matting efficiency.

Exemplarily, the original image and the trimap image are used as input data of the selected matting algorithm, and the matting image can be obtained through image segmentation processing.

Step 203: Acquire a target image in the original image.

Exemplarily, the pixel points marked by the user in the process of generating the trimap map may be recorded, the target object that the user pays attention to is determined according to the pixel points, and the target image of the target object is obtained from the original image.

Step 204: Determine the similarity between the cutout image and the target image, and score the cutout image according to the similarity.

Exemplarily, the cutout image is compared with the target image to determine the similarity between the cutout image and the target image. The correspondence between the similarity and the score interval may be predetermined, so that the matting image is scored according to the similarity. For example, if the similarity exceeds the first threshold, the score of the cutout image is determined to be 80 to 100 points; if the similarity exceeds the second threshold but is less than the first threshold, the score of the cutout image is determined to be 60 to 80 points; If the similarity exceeds the third threshold but is less than the second threshold, the score of the cutout image is determined to be 50 to 60 points; if the similarity exceeds the fourth threshold but is less than the third threshold, the score of the cutout image is determined to be 40 to 50 score; if the similarity exceeds the fifth threshold but is less than the fourth threshold, the cutout image is determined to have a score of 0 to 40.

It should be noted that the method for determining the similarity in the embodiment of the present application may be: reducing the size of the cutout image and the target image, such as reducing the size to 8*8, reducing the image to a set size to remove the image details, only basic information such as structure/light and shade are retained, and image differences caused by different sizes/ratios are discarded. Convert the reduced 8*8 image to 64-level grayscale, that is, the pixels have only 64 colors in total. Calculate the grayscale average of all 64 pixels. Compare the gray value of each pixel in the cutout image with the average value. If the comparison result is greater than or equal to the average value, record the pixel point as 1, and if the comparison result is less than the average value, record the pixel point. is 0; the above comparison results are combined to form a 64-bit integer, which is recorded as the matting image fingerprint. In a similar manner, the target image fingerprint is calculated. Compare the Hamming distance between the fingerprint of the cutout image and the fingerprint of the target image (the Hamming distance between two strings of equal length is the number of different characters in the corresponding positions of the two strings), and calculate the Hamming distance. as the similarity of the two images. That is, if the Hamming distance is large, the difference between the two images is large, and if the Hamming distance is small, the difference between the two images is small, and the two images are more similar.

It should be noted that there are many ways to score the cutout image, which are not limited to the ways listed in the embodiments of the present application. For example, the cutout image may also be scored according to the completeness of the cutout image, and the like. The completeness may be determined according to whether the pixels of adjacent areas in the cutout image are continuous, and may also be determined according to whether the boundary of the cutout image is complete, and so on.

Step 205 , determine whether the score of the cutout image exceeds the set threshold, if yes, go to step 206 , otherwise go to step 201 .

Exemplarily, the score of the cutout image is compared with a set threshold, and when the score of the cutout image is greater than or equal to the set threshold, step 206 is performed. When the score of the cutout image is less than the set threshold, it is determined that the cutout effect of the cutout image is not good, so that it is determined that the trimap image for generating the cutout image does not meet the set conditions, and the execution of the generation based on the trimap image is abandoned. For the operation of training samples, go back to step 201 to obtain the trimap again.

Step 206: Use the original image and the tripartite map as training samples, and generate a training sample set according to the training samples.

Exemplarily, when the trimap image satisfies the set condition, an association relationship between the trimap image and the original image is established as a training sample. After the set number of training samples are acquired, it is determined that the sample collection operation is completed, and the set number of training samples constitutes a training sample set.

Step 207: Train a preset deep learning network based on the training sample set to obtain an image processing model.

It should be noted that the preset deep learning network includes a deep learning network based on image semantic segmentation and a post-processing layer arranged after the output layer of the deep learning network based on image semantic segmentation, and the post-processing layer is the image processing model. the output layer. The segmentation function is set in the post-processing layer, and the category map output by the deep learning network based on image semantic segmentation is processed by the segmentation function, and the trimap map of the original image is obtained. Wherein, the set piecewise function defines the relationship between the gray value interval and the target pixel value. For example, if the pixels with grayscale values from a1 to a2 in the category map are black, the corresponding target pixel value can be (0,0,0), and the pixels with grayscale values from a5 to a6 are white, and the corresponding target pixel value can be (0,0,0). The pixel value can be (255, 255, 255). It should be noted that the above range includes pixels whose color is not black (or white), but close to black (or white), which can be judged according to the gray value to be closer to black or white. Its corresponding target pixel value is set to (0, 0, 0), and for pixels closer to white, its corresponding target pixel value is set to (255, 255, 255). The pixels with grayscale values of a3 to a4 are gray (including pixels that are neither close to black nor close to white), and the corresponding target pixel values may be (192, 192, 192). It can be understood that, there can be many values for the target pixel value, and it is not limited to the pixel values listed in the above examples. For example, the pixel value of cool gray is (128, 138, 135), the pixel value of ivory black is (88, 87, 86) and so on.

Exemplarily, a record in the training sample set is obtained, the original image in the record is input into the deep learning network, and a category map with black and white colors is obtained through the processing of the deep learning network based on image semantic segmentation. The image is passed to the post-processing layer, and the pixel values in the category image are adjusted by the set function to obtain a trimap image with three colors of black, white and gray. The trimap is compared with the trimap obtained in step 201, and the back-propagation algorithm is used to adjust the parameter values of the deep learning network, so that the trimap output by the model is close to the trimap obtained in step 201. After the training is completed, the preset deep learning network is recorded as an image processing model.

In the technical solution of the embodiment of the present application, a matting operation is performed based on the original image and the manually marked trimap image to obtain a matting image; the matting effect is determined by comparing the target image in the original image with the matting image, so that the matting effect is determined. , to determine whether the trimap image meets the set conditions, and only when the set conditions are met, the original image is associated with the trimap image as a training sample, which improves the labeling accuracy of the sample, thereby improving the image processing model trained based on the training sample. Processing accuracy, you can get a better cutout effect. In addition, the image processing model is used to label the original image to automatically generate trimap, which can greatly reduce the workload of labeling.

FIG. 3 is a flowchart of a method for matting provided by an embodiment of the present application. The method may be performed by a matting device, where the device may be implemented by software and/or hardware, and may generally be integrated in a terminal. As shown in Figure 3, the method includes:

Step 310: Obtain the target image to be cut out.

It should be noted that the terminals in the embodiments of the application may include mobile phones, tablet computers, notebook computers, computers, handheld game consoles, smart home appliances, and other devices with operating systems installed. The type of the operating system is not limited in the embodiments of the present application, and may include, for example, an Android (Android) operating system, a Windows (Windows) operating system, and an Apple (ios) operating system, and so on.

The target picture may be a local picture determined based on a user operation or a picture in an Internet platform. Wherein, the user operation may include a touch operation, a voice operation, a gesture operation, an eye gaze operation, and the like. For example, if the user clicks on a certain picture in the local picture library, the picture can be determined as the target picture. For another example, when the user looks at a certain picture on the Internet platform, a prompt message may pop up to ask the user whether the picture needs to be cut out. If the confirmation instruction input by the user is detected, the picture that the user is looking at is determined as the target picture.

Exemplarily, a user operation is detected, a target image to be cutout is determined according to the user operation, and the target image is acquired from a storage location of the target image.

Step 320: Perform labeling processing on the target picture by using an image processing model to obtain a tripartite map of the target picture.

In this embodiment of the present application, the tripartite graph may also be referred to as a trimap graph. The image processing model is a deep learning network trained by the training sample set composed of the original image and the trimap image, and the image processing model is used to label the original image to obtain the trimap image, which can avoid the manual annotation method due to individual cognitive differences. The introduced annotation error greatly improves the annotation accuracy, thereby improving the matting accuracy of matting based on the trimap image.

Exemplarily, the target image is input into the image processing model to obtain a trimap image of the target image.

Step 330: Based on the target image and the three-part map, use a preset matting algorithm to perform a matting process on the target image to obtain a matting image.

It should be noted that it is assumed that the matting algorithm is an image segmentation algorithm for segmenting the foreground from the background (a generalized image segmentation may also include separation between objects of equal status, etc.). The specific algorithm included in the matting algorithm is shown in the above-mentioned embodiment, and details are not repeated here.

Exemplarily, the target image and the trimap image are used as input data for setting the matting algorithm, and the matting image can be obtained through image segmentation processing. Assuming that the target image contains a portrait, the target image and the corresponding trimap image are used as the input data of the matting algorithm. After image segmentation processing, a matting image for the portrait can be obtained, which can achieve a hair-level matting effect.

In the technical solution of the embodiment of the present application, when performing a matting operation, a target image to be matted is obtained; and an image processing model is used to label the target image to obtain a trimap image; then, the target image and the trimap image are used as a design The input data of the matting algorithm is determined, and the matting operation is performed to obtain the matting image of the target image. By adopting the above technical solution, the labeling error introduced by the individual cognitive differences in the manual labeling method can be avoided, and the labeling accuracy is greatly improved, thereby improving the map-cutting accuracy based on the trimap map.

In some examples, after acquiring the target image to be cutout, the method further includes: judging whether the attribute information of the target image matches the sample attribute information of the original image in the training sample set; The processing model performs an operation of labeling the target picture; when there is no match, the attribute information of the target picture is adjusted according to the sample attribute information. Wherein, judging whether the attribute information of the target image matches the sample attribute information of the original image may be judging whether the color space of the target image and the color space of the sample image are the same. The color space includes but is not limited to RGB format, YUV format, HSV format or HIS format, etc. By adopting the above technical solution, since the attribute information of the target image is judged before inputting the target image into the image processing model, it can be ensured that the attribute information of the target image matches the original image in the training sample set, and the introduction of annotations due to the mismatch of attribute information can be avoided. Inaccurate trimap images can be obtained due to errors, and even a situation in which the trimap image cannot be generated due to the labeling failure caused by the mismatch of attribute information, can effectively improve the matting efficiency and matting effect.

4 is a schematic structural diagram of an apparatus for training an image processing model provided by an embodiment of the present application. The apparatus may be implemented by software and/or hardware, and is generally integrated in a terminal or server. The learning network is trained to obtain an image processing model. As shown in Figure 4, the device includes:

A tripartite map obtaining module 410, configured to obtain a tripartite map of the original image;

a sample generation module 420, configured to generate a training sample set according to the original image and the tripartite map;

The model training module 430 is configured to train a preset deep learning network based on the training sample set to obtain an image processing model, wherein the image processing model is used to perform labeling processing on the original image to obtain a tripartite map.

The embodiment of the present application provides a training device for an image processing model, which generates a training sample set according to a plurality of original images and corresponding tripartite graphs by acquiring a tripartite map of an original image; based on the training sample set, a preset deep learning The network is trained to iteratively update the parameter values of the deep learning network. After the training is completed, an image processing model is obtained, and the original image can be labeled and processed through the image processing model to obtain a tripartite map. By adopting the above technical solution, the deep learning network can be trained based on the original image and the corresponding tripartite graph, so that it can automatically label the input original image to obtain the tripartite graph, without manually labeling a large number of hairs. Level data annotation can reduce the workload of annotation and improve the efficiency of image annotation. In addition, the image processing model is used to annotate the original image to avoid errors that may be introduced by manual annotation, which can improve the annotation effect.

Optionally, the sample generation module 420 includes:

A pre-cutting submodule, configured to perform a cut-out process on the original image based on the original image and the three-part image by using a set cut-out algorithm to obtain a cut-out image;

an evaluation sub-module, configured to obtain evaluation information of the cutout image, and determine whether the three-part map satisfies a set condition according to the evaluation information;

The sample generation sub-module is configured to use the original image and the three-part map as training samples, and generate a training sample set according to the training samples if it is determined that the three-part map satisfies the set condition.

Optionally, the evaluation sub-module is specifically used to:

obtaining the target image in the original image;

determining the similarity between the matting image and the target image, and scoring the matting image according to the similarity;

When the score of the cutout image exceeds a set threshold, it is determined that the three-point map satisfies a set condition.

Optionally, the preset deep learning network is a deep learning model in which a post-processing layer is added after the output layer of the deep learning network based on image semantic segmentation, wherein the post-processing layer is used for image semantic segmentation-based deep learning model. The class map with black and white colors output by the deep learning network is subjected to segmentation thresholding to obtain a tripartite map with three colors of black, white and gray.

Optionally, perform segmentation thresholding on the category map with black and white colors output by the deep learning network based on image semantic segmentation, including:

Obtain a set piecewise function, wherein the set piecewise function defines the relationship between the gray value interval and the target pixel value;

A category map with black and white colors output by a deep learning network based on image semantic segmentation is obtained, and the pixel value of the category map is adjusted by using the set segmentation function.

FIG. 5 is a structural block diagram of a matting device provided by an embodiment of the present application. The device can be implemented by software and/or hardware, and can generally be integrated in a terminal, and can perform a matting method based on an image processing model on a target image. Cutout processing. As shown in Figure 5, the device includes:

A target picture obtaining module 510, configured to obtain a target picture to be cut out;

The picture labeling module 520 is configured to perform labeling processing on the target picture through an image processing model to obtain a tripartite map of the target picture, wherein the image processing model is a training sample set composed of the original image and the tripartite map Trained deep learning network;

The matting module 530 is configured to perform matting processing on the target image by adopting a preset matting algorithm based on the target image and the three-part image to obtain a matting image.

An embodiment of the present application provides a matting device, which, when performing a matting operation, acquires a target image to be matted; annotates and processes the target image through an image processing model to obtain a trimap image; and then combines the target image with the trimap image. The image is used as the input data for setting the cutout algorithm, and the cutout operation is performed to obtain the cutout image of the target image. By adopting the above technical solution, the labeling error caused by individual cognitive differences in manual labeling can be avoided, and the labeling accuracy is greatly improved, thereby improving the map-cutting accuracy based on the trimap map.

Optionally, the image matting device further includes:

After acquiring the target image to be cut out, determine whether the attribute information of the target image matches the sample attribute information of the original image in the training sample set;

When matching, perform the operation of labeling the target picture by the image processing model;

When there is no match, the attribute information of the target picture is adjusted according to the sample attribute information.

Embodiments of the present application further provide a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a training method for an image processing model when executed by a computer processor, and the method includes:

Get a tripartite map of the original image;

generating a training sample set according to the original image and the tripartite map;

A preset deep learning network is trained based on the training sample set to obtain an image processing model, wherein the image processing model is used for labeling the original image to obtain a tripartite map.

In addition, an embodiment of the present application also provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a map-out method when executed by a computer processor, and the method includes:

Obtain the target image to be cutout;

An image processing model is used to label the target image to obtain a tripartite map of the target image, wherein the image processing model is a deep learning network trained by a training sample set composed of the original image and the tripartite map;

Based on the target picture and the three-part map, a set cutout algorithm is used to perform cutout processing on the target picture to obtain a cutout image.

storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc. ; non-volatile memory, such as flash memory, magnetic media (eg hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a second, different computer system connected to the first computer system through a network such as the Internet. The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (eg, in different computer systems connected by a network). The storage medium may store program instructions (eg, embodied as a computer program) executable by one or more processors.

Of course, a storage medium containing computer-executable instructions provided by the embodiments of the present application is not limited to the training operation of the image processing model as described above, and the computer-executable instructions can also execute images provided by any embodiment of the present application. Handles related operations in the model's training method.

Certainly, a storage medium containing computer-executable instructions provided by the embodiments of the present application, the computer-executable instructions of the computer-executable instructions are not limited to the above-mentioned image-cutting operations, and can also perform any of the image-cutting methods provided in any of the embodiments of the present application. related operations.

The embodiment of the present application also provides a terminal, in which the training device of the image processing model provided by the embodiment of the present application can be integrated. FIG. 6 is a schematic structural diagram of a terminal according to an embodiment of the present application. As shown in FIG. 6 , the terminal includes a memory 610 and a processor 620 . The memory 610 is used to store computer programs and the like; the processor 620 reads and executes the computer programs stored in the memory 610 . The processor 620 includes a tripartite graph acquisition module 621, a sample generation module 622, a model training module 623, and the like. The processor 620 implements the following steps when executing the computer program: acquiring a tripartite map of the original image; generating a training sample set according to the original image and the tripartite map; The deep learning network is trained to obtain an image processing model, wherein the image processing model is used for labeling the original image to obtain a tripartite map.

In addition, the embodiment of the present application also provides another terminal, and the terminal can integrate the image cutout device provided by the embodiment of the present application. FIG. 7 is a schematic structural diagram of another terminal according to an embodiment of the present application. As shown in FIG. 7 , the terminal includes a memory 710 and a processor 720 . The memory 710 is used to store computer programs and the like; the processor 720 reads and executes the computer programs stored in the memory 710 . The processor 720 includes a target picture acquisition module 721 , a picture labeling module 722 and a map-out module 723 . The processor 720 implements the following steps when executing the computer program: acquiring the target image to be cutout; performing labeling processing on the target image through an image processing model to obtain a tripartite map of the target image, wherein the The image processing model is a deep learning network trained by a training sample set composed of an original image and a tripartite map; based on the target image and the tripartite map, a set-cutting algorithm is used to cut out the target image. , to get the cutout image.

The memories and processors listed in the above examples are all parts of the terminal, and the terminal may also include other components. Taking a smart phone as an example, the possible structure of the above-mentioned terminal will be described. FIG. 8 is a structural block diagram of a smart phone according to an embodiment of the present application. As shown in FIG. 8, the smart phone may include: a memory 801, a central processing unit (Central Processing Unit, CPU) 802 (also known as a processor, hereinafter referred to as CPU), a peripheral interface 803, an RF (Radio Frequency, radio frequency) circuit 805, Audio circuitry 806, speakers 811, touch screen 812, power management chip 808, input/output (I/O) subsystem 809, other input/control devices 810, and external ports 804 through one or more communication buses or signal lines 807 to communicate.

It should be understood that the illustrated smartphone 800 is merely an example of a terminal, and that the smartphone 800 may have more or fewer components than those shown, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The following will describe in detail the smart phone provided in this embodiment with the training device integrated with the image processing model.

Memory 801, which can be accessed by CPU 802, peripheral interface 803, etc., said memory 801 can include high-speed random access memory, and can also include non-volatile memory, such as one or more disk storage devices, flash memory devices , or other volatile solid-state storage devices. A computer program is stored in the memory 801, and a preset file, a preset whitelist, and the like can also be stored.

A peripheral interface 803 that can connect the input and output peripherals of the device to the CPU 802 and the memory 801 .

I/O subsystem 809 , which can connect input and output peripherals on the device, such as touch screen 812 and other input/control devices 810 , to peripherals interface 803 . The I/O subsystem 809 may include a display controller 8091 and one or more input controllers 8092 for controlling other input/control devices 810 . Wherein, one or more input controllers 8092 receive electrical signals from or send electrical signals to other input/control devices 810, which may include physical buttons (push buttons, rocker buttons, etc. ), dial pad, slide switch, joystick, click wheel. Notably, the input controller 8092 can be connected to any of the following: a keyboard, an infrared port, a USB interface, and a pointing device such as a mouse.

The touch screen 812, which is an input interface and an output interface between the user terminal and the user, displays visual output to the user, and the visual output may include graphics, text, icons, videos, and the like.

Display controller 8091 in I/O subsystem 809 receives electrical signals from touch screen 812 or sends electrical signals to touch screen 812 . The touch screen 812 detects the contact on the touch screen, and the display controller 8091 converts the detected contact into interaction with the user interface objects displayed on the touch screen 812, that is, to realize human-computer interaction, and the user interface objects displayed on the touch screen 812 can be run. Icons for games, icons for connecting to the corresponding network, etc. It is worth noting that the device may also include a light mouse, which is a touch-sensitive surface that does not display visual output, or an extension of the touch-sensitive surface formed by a touch screen.

The RF circuit 805 is mainly used to establish the communication between the mobile phone and the wireless network (ie, the network side), and realize the data reception and transmission between the mobile phone and the wireless network. Such as sending and receiving text messages, e-mails, etc. Specifically, the RF circuit 805 receives and transmits RF signals, also known as electromagnetic signals, the RF circuit 805 converts electrical signals into electromagnetic signals or converts electromagnetic signals into electrical signals, and communicates with communication networks and other devices through the electromagnetic signals to communicate. RF circuitry 805 may include known circuitry for performing these functions including, but not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODECs ( COder-DECoder, codec) chip set, subscriber identity module (Subscriber Identity Module, SIM) and so on.

The audio circuit 806 is mainly used to receive audio data from the peripheral interface 803 , convert the audio data into electrical signals, and send the electrical signals to the speaker 811 .

The speaker 811 is used to restore the voice signal received by the mobile phone from the wireless network through the RF circuit 805 to sound and play the sound to the user.

The power management chip 808 is used for power supply and power management for the hardware connected to the CPU 802, the I/O subsystem and the peripheral interface.

The terminal provided by the embodiment of the present application can train a deep learning network based on the original image and the corresponding tripartite graph, so that it can automatically label the input original image to obtain the tripartite graph, without the need for manual annotation. Hair-level data annotation can reduce the workload of annotation and improve the efficiency of image annotation. In addition, the image processing model is used to annotate the original image to avoid errors that may be introduced by manual annotation, which can improve the annotation effect. Therefore, the matting accuracy of matting based on the trimap image is improved.

The training device, matting device, storage medium and terminal of the image processing model provided in the above embodiments can execute the training method or the matting method of the image processing model provided by any embodiment of the present application, and have corresponding functional modules for executing the method. and beneficial effects. For technical details not described in detail in the foregoing embodiments, reference may be made to the training method or the matting method of the image processing model provided by any embodiment of the present application.

Note that the above are only preferred embodiments of the present application and applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present application. The scope is determined by the scope of the appended claims.

Claims (10)

1. a training method of image processing model, is characterized in that, comprises: Get a tripartite map of the original image; generating a training sample set according to the original image and the tripartite map; A preset deep learning network is trained based on the training sample set to obtain an image processing model, wherein the image processing model is used for labeling the original image to obtain a tripartite map, and the preset deep learning network is A deep learning model with a post-processing layer is added after the output layer of the deep learning network based on image semantic segmentation, wherein the post-processing layer is used for the output of the deep learning network based on image semantic segmentation. The class map is segmented and thresholded to obtain a tripartite map with three colors of black, white, and gray. 2. The method according to claim 1, wherein generating a training sample set according to the original image and the tripartite graph, comprising: Based on the original image and the three-part map, a set cutout algorithm is used to perform a cutout process on the original image to obtain a cutout image; Obtain the evaluation information of the cutout image, and determine whether the three-part map meets the set condition according to the evaluation information; If so, use the original image and the tripartite map as training samples, and generate a training sample set according to the training samples. 3. The method according to claim 2, wherein, obtaining the evaluation information of the cutout image, and judging whether the three-part map satisfies a setting condition according to the evaluation information, comprising: obtaining the target image in the original image; determining the similarity between the matting image and the target image, and scoring the matting image according to the similarity; When the score of the cutout image exceeds a set threshold, it is determined that the three-point map satisfies a set condition. 4. method according to claim 1, is characterized in that, carries out segmentation thresholding processing to the class map with black and white two kinds of colors that the deep learning network output based on image semantic segmentation, comprises: Obtain a set piecewise function, wherein the set piecewise function defines the relationship between the gray value interval and the target pixel value; A category map with black and white colors output by a deep learning network based on image semantic segmentation is obtained, and the pixel value of the category map is adjusted by using the set segmentation function. 5. a method for matting, is characterized in that, comprises: Obtain the target image to be cutout; An image processing model is used to annotate the target image to obtain a tripartite map of the target image, wherein the image processing model is a deep learning network trained by a training sample set composed of the original image and the tripartite map. The deep learning network is a deep learning model that adds a post-processing layer after the output layer of the deep learning network based on image semantic segmentation, wherein the post-processing layer is used for the output of the deep learning network based on image semantic segmentation. The category map with two colors of white is subjected to segmentation thresholding to obtain a three-part map with three colors of black, white and gray; Based on the target picture and the three-part map, a set cutout algorithm is used to perform cutout processing on the target picture to obtain a cutout image. 6. method according to claim 5, is characterized in that, after obtaining the target picture to be cutout, also comprises: Judging whether the attribute information of the target image matches the sample attribute information of the original image in the training sample set; When matching, perform an operation of labeling the target picture through the image processing model; When there is no match, the attribute information of the target picture is adjusted according to the sample attribute information. 7. A training device for an image processing model, comprising: Tripartite image acquisition module, used to obtain the tripartite map of the original image; a sample generation module, configured to generate a training sample set according to the original image and the tripartite map; The model training module is used to train a preset deep learning network based on the training sample set to obtain an image processing model, wherein the image processing model is used to perform labeling processing on the original image to obtain a tripartite map, and the preset The set deep learning network is a deep learning model in which a post-processing layer is added after the output layer of the deep learning network based on image semantic segmentation, wherein the post-processing layer is used for the output of the deep learning network based on image semantic segmentation. Segmentation thresholding is performed on the class map with two colors of black, white and gray to obtain a tripartite map with three colors of black, white and gray. 8. a drawing device, is characterized in that, comprises: The target image acquisition module is used to acquire the target image to be cut out; The image labeling module is configured to perform labeling processing on the target image through an image processing model to obtain a tripartite image of the target image, wherein the image processing model is trained on a training sample set composed of the original image and the tripartite image. The deep learning network is a deep learning model that adds a post-processing layer after the output layer of the deep learning network based on image semantic segmentation, wherein the post-processing layer is used for deep learning based on image semantic segmentation. The class map with black and white colors output by the network is subjected to segmentation thresholding to obtain a tripartite map with three colors of black, white and gray; The matting module is configured to perform matting processing on the target image based on the target image and the three-part image by using a preset matting algorithm to obtain a matting image. 9. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the training method of the image processing model according to any one of claims 1-4 is implemented; or, When the program is executed by the processor, the matting method according to claim 5 or 6 is implemented. 10. A terminal, characterized in that it comprises a memory, a processor and a computer program stored on the memory and run on the processor, and when the processor executes the computer program, any one of claims 1-4 is realized. The training method of the image processing model, or, when the processor executes the computer program, the image matting method according to claim 5 or 6 is implemented.

Images