CN109889724B

CN109889724B - Image blurring method and device, electronic equipment and readable storage medium

Info

Publication number: CN109889724B
Application number: CN201910094300.3A
Authority: CN
Inventors: 安世杰; 张渊; 郑文
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2019-01-30
Filing date: 2019-01-30
Publication date: 2020-11-06
Anticipated expiration: 2039-01-30
Also published as: CN109889724A

Abstract

The application relates to an image blurring method, an image blurring device, electronic equipment and a readable storage medium, wherein the method comprises the steps of obtaining a target image; generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number; and performing blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image. The method solves the technical problems that the image blurring method in the related technology has high requirements on hardware equipment, the blurring effect is not ideal enough and the like. The requirements of hardware equipment for reducing image blurring are met, and the technical effect of blurring effect is improved.

Description

Image blurring method and device, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image blurring method and apparatus, an electronic device, and a readable storage medium.

Background

When the single lens reflex is used for photographing, the camera can realize the blurring effect on objects with different depth of field according to the distance of a focus point. However, due to the limitation of hardware devices, electronic devices such as smart phones only have one depth of field for shot pictures, and the blurring effect cannot be achieved, so that shot images are not real enough.

In the related technology, the common method is to predict depth based on binocular vision and to virtualize the background based on segmentation, but the depth of each pixel point in the binocular vision predicted image has high requirements on hardware equipment, and the calculated amount is complex, so that real-time application is difficult to realize; the segmentation-based method ignores the depth of field of the background, and background blurring with low true degree is easy to generate. In addition, the current monocular depth estimation is an ill-posed problem, e.g., the same 2D image may be projected from an infinite number of different 3D scenes. To overcome this difficulty, the classical approach is to use meaningful monocular cues and features such as perspective and texture information, object size, object position and occlusion, etc. However, these methods all require manual design of feature extractors, and the model prediction speed and accuracy are difficult to guarantee. Therefore, the image blurring method in the related technology has the technical problems of high requirements on hardware equipment, unsatisfactory blurring effect and the like.

Disclosure of Invention

In order to overcome the problems in the related art, the present application provides an image blurring method, an image blurring device, an electronic device, and a readable storage medium.

According to a first aspect of embodiments of the present application, there is provided an image method, including:

acquiring a target image;

generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number;

and performing blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image.

Optionally, before the step of generating the depth image of the target image based on the preset convolutional neural network model, the method further includes:

taking the batch size corresponding to the depth prediction model as a unit, and extracting training images of known depth maps from a preset training image set according to the sampling weight corresponding to each scene type;

and training the depth prediction model through the training image until the prediction error of the depth prediction model meets a preset error threshold.

Optionally, the step of extracting a training image of a known depth map from a preset training image set by using the batch size corresponding to the depth prediction model as a unit and the weight corresponding to each scene type includes:

determining the sampling weight of each scene type according to the first number of training images contained in each scene type and the second number of all training images;

determining the extracted probability of each training image under each scene type according to the sampling weight of each scene type;

and extracting the training images of each training from the training image set by taking the batch size as a unit according to the extracted probability of each training image.

Optionally, the blurring processing on each pixel point in the target image according to the depth image to obtain a blurred image of the target image includes:

according to the depth image, generating M fuzzy images with sequentially deepened fuzzy degrees aiming at the target image, wherein M is a natural number;

aiming at each pixel point in the target image, acquiring a first distance between the pixel position depth and the focusing depth of the pixel point according to the depth image;

acquiring a target blurred image corresponding to the pixel point from the M blurred images according to the first distance;

and determining the virtualized pixel value of the pixel point according to the target blurred image.

Optionally, the step of determining a blurred pixel value of the pixel point according to the target blurred image includes:

if the pixel points correspond to a plurality of target blurred images, determining the interpolation weight of each target blurred image according to the first distance;

and carrying out interpolation processing on the pixel points according to the target blurred image and the interpolation weight of the target blurred image to obtain the virtualized pixel value of the pixel points.

Optionally, the depth prediction model sequentially includes a base model, a multi-scale model, a feature fusion layer and a prediction output layer; the loss function of the depth prediction model comprises a Reversed HuBer loss function, a weighted sum of a depth gradient loss function and a normal vector loss function, and the weights of the depth gradient loss function and the normal vector loss function are both smaller than the weight of the Reversed HuBer loss function; the prediction error of the depth prediction model comprises a relative error and/or a root mean square error;

the basic model is used for extracting the characteristics of the current input image;

the multi-scale model is used for extracting a feature map with a preset scale aiming at the current input image;

the feature fusion layer is used for recovering the resolution and the reduced channel number of the current input image and fusing features output by the basic model;

and the prediction output layer is used for acquiring and outputting the pixel position depth of each pixel point in the current input image.

Optionally, each convolution layer in the depth prediction model is a depthwise-pointwise structure.

According to a second aspect of embodiments of the present application, there is provided an image blurring device, including:

a target image acquisition module configured to acquire a target image;

a depth image obtaining module configured to generate a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number;

and the blurring processing module is configured to perform blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image.

Optionally, the apparatus further comprises:

the training image extraction module is configured to extract training images of known depth maps from a preset training image set according to the sampling weight corresponding to each scene type by taking the batch size corresponding to the depth prediction model as a unit;

a model training module configured to train the depth prediction model through the training image until a prediction error of the depth prediction model meets a preset error threshold.

Optionally, the training image extraction module includes:

a sampling weight determination submodule configured to determine a sampling weight for each of the scene types according to a first number of training images included in each of the scene types and a second number of entire training images;

a extracted probability determination sub-module configured to determine an extracted probability of each training image under each scene type according to the sampling weight of each scene type;

and the training image extraction sub-module is configured to extract the training images of each training from the training image set by taking the batch size as a unit according to the extracted probability of each training image.

Optionally, the blurring processing module includes:

a blurred image generation submodule configured to generate M blurred images with sequentially deepened blur degrees for the target image according to the depth image, where M is a natural number;

a first distance obtaining submodule configured to obtain, for each pixel point in the target image, a first distance between a pixel position depth and a focusing depth of the pixel point according to the depth image;

a target blurred image acquisition sub-module configured to acquire a target blurred image corresponding to the pixel point from the M blurred images according to the first distance;

and the blurring pixel value acquisition submodule is configured to determine blurring pixel values of the pixel points according to the target blurred image.

Optionally, the blurred pixel value obtaining sub-module includes:

an interpolation weight determination unit configured to determine an interpolation weight of each of the target blurred images according to the first distance if the pixel point corresponds to a plurality of target blurred images;

and the blurring pixel value acquisition unit is configured to perform interpolation processing on the pixel points according to the target blurred image and the interpolation weights of the target blurred image to obtain blurring pixel values of the pixel points.

According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

a processor;

a memory configured to store processor-executable instructions;

wherein the processor is configured to:

acquiring a target image;

According to a fourth aspect of embodiments herein, there is provided a non-transitory computer readable storage medium having instructions which, when executed by a processor of an electronic device, enable the electronic device to perform a method of image blurring, the method comprising:

acquiring a target image;

According to a fifth aspect of embodiments of the present application, there is provided an application program, which when executed by a processor of an electronic device, enables the electronic device to perform an image blurring method, the method comprising:

acquiring a target image;

The technical scheme provided by the embodiment of the application can have the following beneficial effects: the embodiment of the application acquires a target image; generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number; and performing blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image, so that the requirement of image blurring on hardware equipment can be reduced, and the blurring effect is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, are configured to explain the principles of the application.

FIG. 1 is a flow chart illustrating a method of image blurring according to an exemplary embodiment.

FIG. 2 is a flow chart illustrating another method of image blurring according to an exemplary embodiment.

FIG. 3 is a block diagram illustrating a depth prediction model according to an exemplary embodiment.

Fig. 4a is a schematic diagram illustrating an effect of an image blurring process according to an exemplary embodiment.

FIG. 4b is a schematic diagram illustrating the effect of another image blurring process according to an exemplary embodiment.

FIG. 4c is a schematic diagram illustrating the effect of another image blurring process according to an exemplary embodiment.

FIG. 4d is a schematic diagram illustrating the effect of another image blurring process according to an exemplary embodiment.

Fig. 5 is a block diagram illustrating an image blurring apparatus according to an exemplary embodiment.

Fig. 6 is a block diagram illustrating an image blurring apparatus according to an exemplary embodiment.

FIG. 7 is a block diagram illustrating an apparatus in accordance with an example embodiment.

FIG. 8 is a block diagram illustrating an apparatus in accordance with an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

Fig. 1 is a flowchart illustrating an image blurring method according to an exemplary embodiment, and the image blurring method is configured in a terminal as shown in fig. 1, and includes the following steps.

In step S11, a target image is acquired.

It should be noted that the target image in the embodiment of the present application may include any image that can be subjected to blurring processing, such as a picture that has been taken by a camera, a preview image of the camera, or a frame of picture in a video that has been taken by the camera, and so on. Furthermore, in the embodiment of the present application, the target image may be acquired in any available manner, and the embodiment of the present application is not limited thereto.

In step S12, generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained through training of training images of known depth maps under N scene types, and N is a natural number.

After the target image is obtained, in order to improve the blurring effect of the target image, the depth image of the target image can be obtained, so that the pixel position depth of each pixel point in the target image can be obtained based on the depth image, and then the blurring process is performed on the corresponding pixel point based on the position depth of each pixel point, so that the blurring effect of the target image can be improved. The depth of a pixel position can be understood as the distance from an object corresponding to the corresponding pixel point to a camera plane for shooting a target image.

As described above, in the prior art, the depth of each pixel point in the binocular vision-based prediction image has a high requirement on hardware equipment, and the calculation amount is complex, so that real-time application is difficult to realize. In the embodiment of the present application, in order to improve the depth image obtaining efficiency, the depth image of the target image may be generated through a preset depth prediction model. The depth prediction model is a Convolutional Neural Network (CNN) model obtained by training images of known depth maps in N scene types, and N is a natural number. The specific value of N may be preset as required, and this embodiment of the present application is not limited. In addition, the types specifically included in the scene types and the specific number of the training images may also be preset according to the requirements, and the embodiment of the present application is not limited.

For example, the scene types may be set to include an indoor scene, an outdoor scene, and a manned scene, where N takes a value of 3. The specific number of training images may be set to 807365, and 6737 training images whose depth maps are known per frame may be set to verify the accuracy of the trained depth prediction model.

After the training images are obtained, the training images can be used as the input of the depth prediction model, and the depth maps corresponding to the corresponding training images are used as the output of the depth prediction model, so as to train parameters in the depth prediction model.

Note that, the depth image of the target image is acquired by the depth prediction model, and the number of cameras included in the photographing apparatus is not limited in any way.

In step S13, blurring each pixel point in the target image according to the depth image to obtain a blurred image of the target image.

After the depth image of the target image is obtained, blurring processing can be performed on each pixel point in the target image based on the corresponding depth image, so that a blurred image of the target image can be obtained. At this time, according to the difference of the depths of the pixel positions, the blurring processing degrees of the corresponding pixel points can be correspondingly different, so that the blurring processing of different programs is performed on the pixel points at different positions in the target image, and further the blurring image of the target image is obtained.

The correspondence between the pixel position depth and the blurring program may be preset according to a requirement, and the embodiment of the present application is not limited thereto.

The embodiment of the application acquires a target image; generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number; and performing blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image, so that the requirement of image blurring on hardware equipment can be reduced, and the blurring effect is improved.

Fig. 2 is a flowchart illustrating an image blurring method according to an exemplary embodiment, where the image blurring method is configured in a terminal as shown in fig. 2, and includes the following steps.

In step S21, a target image is acquired.

And step S22, taking the batch size corresponding to the depth prediction model as a unit, and extracting the training images of the known depth maps from a preset training image set according to the sampling weight of the training images under each scene type.

In practical application, training images that can be input by a constructed depth prediction model at one time are limited and can be defined as batch size (batch size), when the training images are set, the number of the training images contained in different scene types may not be consistent, and if a batch size picture is randomly selected from all the training images to train the depth prediction model according to a conventional method, the problem of unbalanced training samples may occur, so that the trained depth prediction model is inaccurate.

Therefore, it is necessary to set appropriate sampling weights for different scene types in order to solve the problem of the imbalance in the number of training images between different scene types. And further, the training images of the known depth maps are extracted from a preset training image set according to the sampling weight corresponding to each scene type by taking the batch size corresponding to the depth prediction model as a unit. The weight corresponding to each scene type may be preset according to a requirement, and the embodiment of the present application is not limited thereto. The specific value of batch size also can preset according to the demand, does not restrict this application embodiment yet. For example, the batch size may be set to 128 based on a model size of 120M (megabytes) and a limit of 11G (gigabytes) of display memory for computing resources.

For example, if the scene types include three types, i.e., an indoor scene, an outdoor scene, and a manned scene, and the weights of the training images in the respective scene types are a, b, and c in turn, the numbers of the training images included in the respective scene types are d1, d2, and d3, respectively. Then the probabilities extracted for each training image for the three scene types are, in order, a/(a × d1+ b × d2+ c × d3), b/(a × d1+ b × d2+ c × d3), and c/(a × d1+ b × d2+ c × d 3). Then if the batch size is 128, the number of training images extracted from each scene type is respectively 128 × d1 × a/(a × d1+ b × d2+ c × d3), 128 × d2 × b/(a × d1+ b × d2+ c d3), 128 × d3 × c/(a × d1+ b × d2+ c × d 3). At this time, if it is necessary to ensure the uniformity of the training images extracted each time, it is possible to ensure that the number of training images extracted from each scene type is the same, i.e., 128 × d1 × a/(a × d1+ b × d2+ c × d3) ═ 128 × d2 × b/(a × d1+ b d2+ c d3) ═ 128 × d3 × c/(a × d1+ b × d2+ c 3). Further, it is assumed that d1 a d2 b d3 c is required.

Of course, in this embodiment of the application, if the training images in each scene type are respectively set in different training image sets, then the sample size/N training images may be respectively extracted from the training image set corresponding to each scene type each time according to the specific value of the sample size, and at this time, the sampling weights of the training images in each scene type may not need to be set.

Optionally, in this embodiment of the application, the step S22 further may include:

step S221, determining the sampling weight of each scene type according to the first number of training images contained in each scene type and the second number of all training images;

as can be seen from the above analysis, if the extracted training images are more balanced in each scene type as much as possible, the weight corresponding to each scene type may be set to satisfy d1 a-d 2 b-d 3 c, and then the sampling weight for each scene type may be determined according to the first number of training images included in each scene type and the second number of all training images. The corresponding relationship between the sampling weight corresponding to each scene type and the first number of training images included in the scene type, as well as the second number of all training images, may be preset according to the requirement, and the embodiment of the present application is not limited thereto.

For example, for the types of scenes described above: the first number of training images contained in the indoor scene, the outdoor scene, the manned scene, and each scene type is d1, d2, and d3, respectively. Then the sampling weight a of the indoor scene may be set to (d1+ d2+ d3)/d1, the sampling weight b of the outdoor scene may be set to (d1+ d2+ d3)/d2, and the sampling weight c of the human scene may be set to (d1+ d2+ d3)/d 3.

Step S222, determining the extracted probability of each training image under the scene type according to the sampling weight of each scene type;

after determining the sampling weight for each scene type, the extracted probability for each training image for the respective scene type may then be determined based further on the sampling weight for each of the scene types.

For example, for the types of scenes described above: the first number of training images contained in an indoor scene, an outdoor scene, a manned scene, and each scene type is d1, d2, and d3, respectively, assuming that the sampling weights of each scene type are a, b, and c, respectively, in turn. Then the extracted probability for each training image in the indoor scene type may be a/(a × d1+ b × d2+ c × d3), the extracted probability for each training image in the outdoor scene type may be b/(a × d1+ b × d2+ c × d3), and the extracted probability for each training image in the manned scene type may be c/(a × d1+ b × d2+ c × d 3). The sampling weights under all the scene types are sequentially substituted into the extracted probability, the extracted probability of each training image under the indoor scene type is 1/(3 × d1), the extracted probability of each training image under the outdoor scene type is 1/(3 × d2), and the extracted probability of each training image under the manned scene type is 1/(3 × d 3).

Therefore, in the embodiment of the present application, the extracted probability of each training image in each scene type may also be directly set to be 1/(N × d), where N is the total number of scene types, and d is the first number of training images included in the corresponding scene type. At this time, the extracted probability of each training image in the corresponding scene type may be determined directly according to the first number of training images included in each scene type and the total number of scene types, without setting the sampling weight of each scene type first.

And step S223, extracting the training images of each training from the training image set by taking the batch size as a unit according to the extracted probability of each training image.

At this time, based on the extracted probability of each training image, the training images required for each training can be extracted from the training image set in units of the batch size of the depth prediction model. At this time, because the extracted probabilities of the training images in different scene types are related to the first number of the training images included in the corresponding scene type, the smaller the numerical value of the first number is, the larger the extracted probability of the training images in the corresponding scene type is, so that the balance of the extracted training images can be improved, and the training effect of the depth prediction model can be further improved.

And step S23, training the depth prediction model through the training image until the prediction error of the depth prediction model meets a preset error threshold value.

After the training image is extracted each time, the depth preset model can be trained through the currently extracted training image until the prediction error of the depth prediction model meets the preset error threshold. The preset error threshold value may be preset according to a requirement, and the embodiment of the present application is not limited. For example, the preset error threshold may be set to 100% or less, and so on.

At this time, after the depth prediction model is trained on the basis of the currently extracted training image, the prediction error of the currently trained depth prediction model is obtained by using another batch of training images of known depth images, and then whether the current prediction error meets a preset error threshold value is compared, if the current prediction error meets the preset error threshold value, the depth prediction model is not trained any more, and the depth image of the target image can be obtained on the basis of the depth prediction model obtained by the current training; if the current prediction error does not satisfy the preset error threshold, the process may return to step S22, or return to step S223, and extract the training image and train the depth prediction model again until the prediction error of the depth prediction model satisfies the preset error threshold.

The calculation mode of the prediction error of the depth prediction model may be preset according to the requirement, and the embodiment of the present application is not limited. For example, the prediction Error may be set to include, but is not limited to, a relative Error (Mean relative Error), a Root Mean Squared Error (Root Mean Squared Error), and so on.

Step S24, generating a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained through training of training images of known depth maps under N scene types, and N is a natural number.

Step S25, according to the depth image, M blurred images with sequentially increased blur degrees are generated for the target image, where M is a natural number.

In order to facilitate blurring of each pixel point in the target image according to the depth of the pixel position, M blurred images with sequentially deepened blur degrees may be generated for the target image according to the depth image of the target image, where a value of M may be preset according to a requirement, and this embodiment of the present application is not limited. Furthermore, in the embodiments of the present application, the blurred image may be generated in any available manner, and the embodiments of the present application are not limited thereto. For example, M blurred images whose blur degrees are sequentially deepened may be generated for the target image using gaussian blur, or the like. The specific blurring degrees of the M blurred images may also be preset according to requirements, and the embodiment of the present application is not limited.

Step S26, aiming at each pixel point in the target image, obtaining the pixel position depth and the first distance of the focusing depth of the pixel point according to the depth image.

In practical application, when the shooting is performed on the same scene, the focusing depths are different, and the blurring degrees of the shot images can be correspondingly different, so that the blurring images with incompletely consistent blurring effects can be generated on the same target image according to different focusing depths. The focusing depth can be set by a photographer in a user-defined manner according to requirements, and the embodiment of the application is not limited. Then, in order to generate different blurring effects based on the current focusing depth, a first distance between the pixel position depth of each pixel point and the currently set focusing depth may be obtained according to the depth image of the target image for each pixel point in the target image.

Step S27, according to the first distance, acquiring a target blurred image corresponding to the pixel point from the M blurred images.

And then, according to the first distance, acquiring a target blurred image corresponding to each pixel point in the target image from the M blurred images generated in advance. The corresponding relationship between the blurred image and the distance value may be preset according to the blur degree of each blurred image, and the embodiment of the present application is not limited.

For example, if 6 blurred images are generated in advance, the images are sorted in the order of P1, P2, P3, P4, P5, and P6 in the order of successively deepening the blur program. Each pixel point of the target image can select at least one fuzzy degree in the 6 fuzzy images. If the absolute value of the difference between the pixel position depth and the focusing depth is set, that is, the pixel point with the first distance of 1 meter corresponds to P1, the pixel point with the first distance of 2 meters corresponds to P2, and the pixel points with the first distance of more than 1 meter and less than 2 meters correspond to P1 and P2, and so on. Then if the focusing depth is 3m, the pixel points with depth values of 2 and 4m are P1, the pixels with depths of 1m and 5m are P2, and the depth is 1.5, then P1 and P2 can be selected to be obtained on average. Therefore, the 3m pixels are clear, and the pixels at other positions can be sequentially selected to be more fuzzy according to the distance from the 3 m.

And step S28, determining the blurring pixel value of the pixel point according to the target blurred image.

After the target blurred image corresponding to each pixel point in the target image is determined, the blurred pixel value of the corresponding pixel point can be determined according to the target blurred image corresponding to each pixel point. The specific correspondence between the blurred pixel value and the target blurred image may be preset according to requirements, and the embodiment of the present application is not limited thereto.

For example, if a certain pixel point pixel _1 corresponds to a target blurred image, the pixel value at the position of the corresponding pixel point pixel _1 in the target blurred image can be directly obtained as the virtualized pixel value of the corresponding pixel point pixel _ 1; if a certain pixel point pixel _2 corresponds to a plurality of target blurred images, the weighted sum of the pixel values at the position of the corresponding pixel point pixel _1 in the plurality of target blurred images can be obtained and used as the virtualized pixel value of the corresponding pixel point pixel _1, and at this time, the weight of each target blurred image can be preset according to requirements, and the embodiment of the application is not limited.

Optionally, in this embodiment of the application, the step S28 further may include:

step S281, if the pixel point corresponds to multiple target blurred images, determining an interpolation weight of each target blurred image according to the first distance;

as described above, in practical applications, there may be pixel points corresponding to multiple target blurred images, and in order to improve the accuracy of determining the blurred pixel values of the corresponding pixel points according to the multiple corresponding target blurred images, the interpolation weight of each target blurred image may be determined according to the first distance between the depth of the pixel position of the corresponding pixel point and the depth of focus. The corresponding relationship between the interpolation weight and the first distance may be preset according to a requirement, and the embodiment of the present application is not limited.

For example, for the 6 blurred images, the blurred images are sequentially sorted in the order of increasing depth according to the blurring procedure as P1, P2, P3, P4, P5 and P6, the pixel point with the first distance of 1 meter corresponds to P1, the pixel point with the first distance of 2 meters corresponds to P2, the pixel points with the first distance of more than 1 meter and less than 2 meters correspond to P1 and P2, and so on. For the pixel points corresponding to the multiple target blurred images, the interpolation weight of the target blurred image with higher blurring degree corresponding to the corresponding pixel point may be set as the difference between the first distance corresponding to the corresponding pixel point and the maximum positive integer smaller than the first distance, and the interpolation weight of the target blurred image with lower blurring degree corresponding to the corresponding pixel point is the absolute value of the difference between the first distance corresponding to the corresponding pixel point and the minimum positive integer larger than the first distance. Assuming that the first distance corresponding to a certain pixel point is 1.7, the target blurred images corresponding to the pixel point are P1 and P2, and the interpolation weight of P1 is 0.3, and the difference weight of P2 is 0.7.

Step S282, performing interpolation processing on the pixel point according to the target blurred image and the interpolation weight of the target blurred image, to obtain a blurred pixel value of the pixel point.

For the pixel points corresponding to the multiple target blurred images, after the interpolation weights of the multiple target blurred images corresponding to the pixel points are obtained, interpolation processing can be performed on the pixel points according to the target blurred images and the interpolation weights of the target blurred images to obtain the virtualized pixel values of the pixel points.

Specifically, the weighted summation may be performed on the pixel values corresponding to the corresponding pixel points in the corresponding target blurred images according to the interpolation weights corresponding to the target blurred images, so as to obtain RGB (Red, Green, Blue, Red, Green, Blue) values of the corresponding pixel points as the virtualized pixel values thereof, and so on.

Optionally, as shown in fig. 3, in the embodiment of the present application, the depth Prediction Model sequentially includes a Base Model (Base Model), a Multi-Scale Model (Multi-Scale Model), a Feature fusion layer (Feature Fuse layer), and a Prediction output layer (Prediction layer); the loss function of the depth prediction model comprises a Reversed HuBer loss function, a weighted sum of a depth gradient loss function and a normal vector loss function, and the weights of the depth gradient loss function and the normal vector loss function are both smaller than the weight of the Reversed HuBer loss function; the prediction Error of the depth prediction model comprises a Relative Error (Mean Relative Error) and/or a Root Mean square Error (Root Mean squared Error). Also, Conv2d shown in fig. 3 represents a convolutional layer.

The basic Model is used for extracting the features of the current input image so as to provide the features for the Multi-Scale Model; and the features may include, but are not limited to, features of the corresponding image from a bottom layer to a top layer, for example, the bottom layer features may include edges, corners, texture, color information, and the like of the image; the middle layer features may include geometric information of the image, such as structures of circles, rectangles, triangles, etc.; the high-level features may include semantic information of the image, such as people, buildings, sky, etc.

And the multi-scale model is used for extracting a feature map with a preset scale aiming at the current input image. The preset scale can be preset according to requirements, and the embodiment of the application is not limited. Extracting Feature maps of different scales, mainly in order to consider local information and global information in corresponding images, wherein the relative relation of the point on the original image on the whole image is recorded in each pixel position in the Feature maps, so that the local Feature information and the global Feature information can be provided for the Feature Fuse layer and the Prediction layer.

For example, the depth prediction Model shown in FIG. 3, in a Multi-Scale Model, the feature map sizes of the four convolution modules may be set to 16x16, 8x8, 4x4 and 1x1, on a large scale, because the field of view corresponding to the original image is small and the picture is reduced, which is equivalent to that the picture is far away from human eyes, the field of view is larger, and on the contrary, the field of view is smaller, so it is mainly a local feature, and 8x8 and 4x4 are also local features on a larger receptive field, i.e., each pixel sees the larger range of the original image, 1x1 is a global feature, which can reflect the (relative) distance relationship of each position of the local area, for example, the distance relationship between the head and the tail of a bed, the global features can measure the distance relationship between local features at various positions in an image, such as the distance relationship between every furniture in a room, which is very important for the prediction of depth estimation.

The feature fusion layer is used for recovering the resolution and the reduced channel number of the current input image and fusing features output by the basic model, so that the features from the bottom layer to the high layer can be considered.

Because the number of channels of the Feature map becomes 2 times thicker and the length and width of the channel becomes half smaller for each convolution in the Base Model process, the number of channels will be half smaller and the length and width will be half smaller for each convolution in the Feature Fuse layer, but more information will be lost in the channel size reduction and amplification processes, and the Feature maps with the same size, i.e. the number of channels, in the previous corresponding layers will be fused. The specific fusion mode can be preset according to the requirement, and the embodiment of the application is not limited. For example, an additive fusion method can be adopted, so that the calculation amount can be reduced. In the process, the part enlarged from the small graph is from the high-level feature, and the added corresponding layer is the low-level feature, so that the method simultaneously utilizes the low-level feature and the high-level feature, thereby ensuring that useful information is not lost. The closer to the input is the low-layer feature, and the closer to the output is the high-layer feature, the lower-layer feature is fused at the output end, so that the prediction precision can be improved.

And the prediction output layer is used for acquiring the pixel position depth of each pixel point in the current input image so as to obtain and output a depth image of the target image.

In addition, because the convolution kernel of the normal 3x3 convolution has a size of 3x3xInxOut, where In is an input channel and Out is an output channel, the calculated amount is 3x3xInxOut, but In the embodiment of the present application, each convolution layer In the depth prediction model may be a depthwise-pointwise structure, and the structure of the depth prediction model at this time is to perform a channel-by-channel 3x3 convolution first, and then perform a 1x1 normal convolution to expand the number of Out channels, where the calculated amount is only 3x3xIn +1x1xOut, and the calculated amount may be generally reduced by 8-9 times.

In the embodiment of the present application, the depth prediction model may also be configured to update parameters of the model by using a random gradient descent method, and of course, any other available method may also be used to update parameters of the model, which is not limited in the embodiment of the present application.

For example, for the trained depth prediction model, assuming that a target image as shown on the leftmost side in fig. 3 is input, the depth prediction model may output a depth image as shown on the rightmost side of fig. 3.

The specific application scenarios of the embodiment of the application can be as follows:

(1) when a user takes a picture, the depth image of the corresponding picture is obtained by using the depth prediction model.

(2) According to the depth image, a user selects a target area to be focused to form a corresponding large aperture image, and different large aperture effects are formed by different focuses as shown in FIGS. 4a and 4 b;

(3) after the focusing position is determined, the user can form pictures with different depth of field effects by adjusting the aperture value, as shown in fig. 4a and 4 c;

(4) after the focusing position is determined, the user can also form pictures with different clear ranges by adjusting the size of the focusing point, as shown in fig. 4a and 4 d.

The Focus and the orange frame represent focusing positions, the focal length range is 0-1, and the values can change when the positions change; blur represents the size of an aperture value, the range of the aperture is 0-1, the larger the value is, the larger the aperture is, the shallower the depth of field is, and the more fuzzy the background is; range represents the size of a focusing frame, the Range of the focusing frame is 0-1, the larger the value is, the larger the focusing area is, and the larger the clear Range is.

Moreover, in the embodiment of the present application, a batch size corresponding to the depth prediction model may be used as a unit, and a training image of a known depth map may be extracted from a preset training image set according to a sampling weight corresponding to each scene type; and training the depth prediction model through the training image until the prediction error of the depth prediction model meets a preset error threshold. Determining the sampling weight of each scene type according to the first number of training images contained in each scene type and the second number of all training images; determining the extracted probability of each training image under each scene type according to the sampling weight of each scene type; and extracting the training images of each training from the training image set by taking the batch size as a unit according to the extracted probability of each training image. The depth prediction model sequentially comprises a basic model, a multi-scale model, a feature fusion layer and a prediction output layer; the loss function of the depth prediction model comprises a ReveredHuBer loss function, a weighted sum of a depth gradient loss function and a normal vector loss function, and the weights of the depth gradient loss function and the normal vector loss function are both smaller than the weight of the Revered HuBer loss function; the prediction error of the depth prediction model comprises a relative error and/or a root mean square error. Each convolution layer in the depth prediction model is of a depthwise-pointwise structure. Therefore, the accuracy of the depth prediction model can be further improved, and the image blurring effect is further improved.

In addition, in the embodiment of the application, M blurred images with sequentially deepened blur degrees may be generated for the target image according to the depth image, where M is a natural number; aiming at each pixel point in the target image, acquiring a first distance between the pixel position depth and the focusing depth of the pixel point according to the depth image; acquiring a target blurred image corresponding to the pixel point from the M blurred images according to the first distance; and determining the virtualized pixel value of the pixel point according to the target blurred image. If the pixel points correspond to a plurality of target blurred images, determining the interpolation weight of each target blurred image according to the first distance; and carrying out interpolation processing on the pixel points according to the target blurred image and the interpolation weight of the target blurred image to obtain the virtualized pixel value of the pixel points. Thereby, the image blurring effect can be further improved.

Fig. 5 is a block diagram illustrating an image blurring apparatus according to an exemplary embodiment. Referring to fig. 5, the apparatus includes a target image acquisition module 31, a depth image acquisition module 32, and a blurring processing module 33.

A target image acquisition module 31 configured to acquire a target image;

a depth image obtaining module 32 configured to generate a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number;

the blurring processing module 33 is configured to perform blurring processing on each pixel point in the target image according to the depth image, so as to obtain a blurring image of the target image.

Fig. 6 is a block diagram illustrating an image blurring apparatus according to an exemplary embodiment. Referring to fig. 6, the apparatus includes a target image acquisition module 41, a training image extraction module 42, a model training module 43, a depth image acquisition module 44, and a blurring processing module 45.

A target image acquisition module 41 configured to acquire a target image;

a training image extraction module 42, configured to extract training images of known depth maps from a preset training image set according to the sampling weight corresponding to each scene type by using the batch size corresponding to the depth prediction model as a unit;

optionally, in this embodiment of the present application, the training image extraction module 42 further includes:

A model training module 43 configured to train the depth prediction model through the training image until a prediction error of the depth prediction model meets a preset error threshold.

A depth image obtaining module 44 configured to generate a depth image of the target image based on a preset depth prediction model; the depth prediction model is a convolutional neural network model obtained by training images of known depth maps under N scene types, and N is a natural number;

and a blurring processing module 45 configured to perform blurring processing on each pixel point in the target image according to the depth image, so as to obtain a blurring image of the target image.

Optionally, in this embodiment of the application, the blurring processing module 45 may further include:

a blurred image generation sub-module 451 configured to generate M blurred images with sequentially deepened blur degrees for the target image, M being a natural number, according to the depth image;

a first distance obtaining sub-module 452 configured to, for each pixel point in the target image, obtain a first distance between a pixel position depth and a focusing depth of the pixel point according to the depth image;

a target blurred image obtaining sub-module 453 configured to obtain a target blurred image corresponding to the pixel point from the M blurred images according to the first distance;

a virtualized pixel value obtaining submodule 454 configured to determine a virtualized pixel value of the pixel point according to the target blurred image.

Optionally, in this embodiment of the application, the blurred pixel value obtaining sub-module 454 further includes:

Optionally, in this embodiment of the present application, the depth prediction model sequentially includes a base model, a multi-scale model, a feature fusion layer, and a prediction output layer; the loss function of the depth prediction model comprises a Reversed HuBer loss function, a weighted sum of a depth gradient loss function and a normal vector loss function, and the weights of the depth gradient loss function and the normal vector loss function are both smaller than the weight of the Reversed HuBer loss function; the prediction error of the depth prediction model comprises a relative error and/or a root mean square error;

Optionally, in an embodiment of the present application, each convolution layer in the depth prediction model is a depthwise-pointwise structure.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

FIG. 7 is a block diagram illustrating an apparatus 500 configured for image blurring, according to an example embodiment. For example, the apparatus 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 7, the apparatus 500 may include one or more of the following components: a processing component 502, a memory 504, a power component 506, a multimedia component 508, an audio component 510, an input/output (I/O) interface 512, a sensor component 514, and a communication component 516.

The processing component 502 generally controls overall operation of the device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 502 may include one or more processors 520 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 502 can include one or more modules that facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

The memory 504 is configured to store various types of data to support operation at the device 500. Examples of such data include instructions for any application or method configured to operate on the apparatus 500, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 504 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 506 provides power to the various components of the device 500. The power components 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 500.

The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front facing camera and/or a rear facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 500 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 510 is configured to output and/or input audio signals. For example, audio component 510 includes a Microphone (MIC) configured to receive external audio signals when apparatus 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 504 or transmitted via the communication component 516. In some embodiments, audio component 510 further includes a speaker configured to output audio signals.

The I/O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor component 514 includes one or more sensors configured to provide various aspects of status assessment for the apparatus 500. For example, the sensor assembly 514 may detect an open/closed state of the device 500, the relative positioning of the components, such as a display and keypad of the apparatus 500, the sensor assembly 514 may also detect a change in the position of the apparatus 500 or a component of the apparatus 500, the presence or absence of user contact with the apparatus 500, orientation or acceleration/deceleration of the apparatus 500, and a change in the temperature of the apparatus 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, configured for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 516 is configured to facilitate communication between the apparatus 500 and other devices in a wired or wireless manner. The apparatus 500 may access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components configured to perform the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 504 comprising instructions, executable by the processor 520 of the apparatus 500 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

FIG. 8 is a block diagram illustrating an apparatus 600 configured for image blurring, according to an example embodiment. For example, the apparatus 600 may be provided as a server. Referring to fig. 8, the apparatus 600 includes a processing component 622 that further includes one or more processors, and memory resources, represented by memory 632, configured to store instructions, such as applications, that are executable by the processing component 622. The application programs stored in memory 632 may include one or more modules that each correspond to a set of instructions. Further, the processing component 622 is configured to execute instructions to perform the above-described methods.

The apparatus 600 may also include a power component 626 configured to perform power management of the apparatus 600, a wired or wireless network interface 650 configured to connect the apparatus 600 to a network, and an input/output (I/O) interface 658. The apparatus 600 may operate based on an operating system stored in the memory 632, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

It should be noted that the execution subject of the present application may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.; or may be a server. When the electronic device is a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, etc., as shown in fig. 5. When the electronic device is a server, as shown in fig. 6.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. An image blurring method, comprising:

acquiring a target image;

performing blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image;

the depth prediction model sequentially comprises a basic model, a multi-scale model, a feature fusion layer and a prediction output layer; the loss function of the depth prediction model comprises a Reversed HuBer loss function, a weighted sum of a depth gradient loss function and a normal vector loss function, and the weights of the depth gradient loss function and the normal vector loss function are both smaller than the weight of the Reversed HuBer loss function; the prediction error of the depth prediction model comprises a relative error and/or a root mean square error;

2. The method of claim 1, further comprising, before the step of generating the depth image of the target image based on the preset convolutional neural network model:

3. The method according to claim 2, wherein the step of extracting the training image of the known depth map from the preset training image set by using the batch size corresponding to the depth prediction model as a unit and the weight corresponding to each scene type comprises:

4. The method according to claim 1, wherein the step of blurring each pixel point in the target image according to the depth image to obtain a blurred image of the target image includes:

aiming at each pixel point in the target image, acquiring a first distance between the pixel position depth and the focusing depth of the pixel point according to the depth image, wherein the first distance is the absolute value of the difference between the pixel position depth and the focusing depth;

5. The method of claim 4, wherein the step of determining the blurred pixel values of the pixel points from the target blurred image comprises:

6. The method of claim 1, wherein each convolutional layer in the depth prediction model is a depthwise-pointwise structure.

7. An image blurring apparatus, comprising:

a target image acquisition module configured to acquire a target image;

the blurring processing module is configured to perform blurring processing on each pixel point in the target image according to the depth image to obtain a blurring image of the target image;

8. The apparatus of claim 7, further comprising:

9. The apparatus of claim 8, wherein the training image extraction module comprises:

10. The apparatus of claim 7, wherein the blurring module comprises:

a first distance obtaining sub-module configured to obtain, for each pixel point in the target image, a first distance between a pixel position depth and a focusing depth of the pixel point according to the depth image, where the first distance is an absolute value of a difference between the pixel position depth and the focusing depth;

11. The apparatus of claim 10, wherein the blurred pixel value obtaining sub-module comprises:

12. The apparatus of claim 7, wherein each convolutional layer in the depth prediction model is a depthwise-pointwise structure.

13. An electronic device, comprising:

a processor;

a memory configured to store processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the image blurring method of any one of claims 1-6.

14. A non-transitory computer readable storage medium, instructions in which, when executed by a processor of an electronic device, enable the electronic device to perform the image blurring method of any one of claims 1-6.