CN111797797A - Face image processing method based on grid deformation optimization, terminal and storage medium - Google Patents

Abstract

The invention discloses a face image processing method, terminal and storage medium based on grid deformation optimization. The method includes: acquiring a posture face image, acquiring a first feature lattice of the posture face image and corresponding The second feature lattice of the predicted frontal face image of the The first grid network and the second grid network corresponding to the predicted frontal image; according to the second grid network and the second feature lattice, the positions of the first feature points and the The first mesh network is optimized to process the pose face image into a frontal face image. The present invention realizes that the gesture face image is converted into a front face image, which enables the face recognition technology to recognize the gesture face image and improves the performance of the face recognition system.

Description

Face image processing method, terminal and storage medium based on mesh deformation optimization

technical field

The present invention relates to the technical field of terminals, in particular to a face image processing method, terminal and storage medium based on grid deformation optimization.

Background technique

Face recognition technology is widely used in various fields, but the current face recognition technology is often limited to the recognition of frontal faces (faces facing straight ahead), which makes the recognition efficiency of the face recognition system low.

Therefore, the existing technology still needs to be improved and improved.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects of the prior art, the present invention provides a face image processing method, terminal and storage medium based on mesh deformation optimization, aiming at solving the problem of low recognition efficiency of face recognition technology in the prior art.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is as follows:

A first aspect of the present invention provides a face image processing method based on mesh deformation optimization, the method comprising:

Obtain the first feature lattice of the pose face image and the second feature lattice of the predicted frontal face image corresponding to the pose face image, wherein the first feature lattice includes each first feature point, and the second feature The lattice includes each second feature point;

respectively constructing a first grid network corresponding to the first feature lattice and a second grid network corresponding to the second feature lattice;

Optimizing the first grid network according to the second grid network, the second feature lattice and the first feature lattice;

The pose face image is converted into a target front face image according to the optimized first mesh network.

The described face image processing method based on grid deformation optimization, wherein, obtaining the second feature lattice of the predicted frontal face image corresponding to the posture face image comprises:

constructing a face shape database, and obtaining the feature vector of the face shape database;

The second feature lattice is obtained according to the first preset formula,

Wherein, the first preset formula is:

为所述人脸形状数据库的平均形状，O为所述姿态脸图像的人脸形状，n₀为常数，n₀-1表示去掉的特征向量的个数。Wherein, Q ₀ is the vector representation of the second feature lattice, E _i represents the ith feature vector of the face shape database,

is the average shape of the face shape database, O is the face shape of the pose face image, n ₀ is a constant, and n ₀ -1 represents the number of removed feature vectors.

The described face image processing method based on grid deformation optimization, wherein the first grid network corresponding to the first feature lattice and the second grid network corresponding to the second feature lattice are respectively constructed Previously included:

The number of the first feature points in the first feature lattice and the number of the second feature points in the second feature lattice are expanded.

The described face image processing method based on grid deformation optimization, wherein the respectively constructing the first grid network corresponding to the posture face image and the second grid network corresponding to the predicted frontal face image comprises:

respectively constructing the first mesh network and the second mesh network according to a second preset formula;

Wherein, the second preset formula is:

P _i+1,j +P _i-1,j +P _i,j+1 +P _i,j-1 -4P _i,j =0

_i =0,...,Nu; j=0,..., _Nv

Among them, P _i,j is a grid point in the i-th row and j-th column in the grid, N _u +1 is the number of rows of the grid, and N _v +1 is the number of columns of the grid.

The method for processing face images based on grid deformation optimization, wherein the first feature is analyzed according to the second grid network, the second feature lattice and the first feature lattice. Optimizing the location of points and the first mesh network includes:

Perform initial optimization on the first mesh network according to a third preset formula;

Re-optimize the first grid network on which the initial optimization has been performed according to the first optimization function, the second optimization function and the third optimization function;

Wherein, the third preset formula is:

Wherein, P _i,j , P′ _i,j are the grid points of the first grid network and the second grid network, respectively, Q _t , Q′ _t are the grid points from the first grid network The t-th first feature point in the grid starting from the grid point P _i,j and the t-th second feature point in the grid starting from the grid point P′ _i,j in the second grid network Feature points;

The first optimization function, the second optimization function and the third optimization function are respectively constructed based on smoothness, translation invariance and left-right symmetry of the human face.

The described face image processing method based on grid deformation optimization, wherein, the first optimization function is:

E _TPS (z(P _i,j ))=(zx″ _u,u ) ² +2(zx″ _u,v ) ² +(zx″ _v,v ) ² +(zy″ _u,u ) ² +2 (zy″ _u,v ) ² +(zy″ _v,v ) ²

Among them, z(P _i,j )=(zx,zy), represents the offset of the P i, _j grid point, zx is the offset of the Pi _,j grid point in the u direction, and zy is P The offset of _{i, j} grid points in the v direction, zx″ _{u, v} represents the second-order partial derivative of zx with respect to the u and v directions;

The second optimization function is:

Wherein, z(Q _t )=Q' _t -Q _t , which represents the translation vector between Q _t and Q'_t;

The third optimization function is:

分别表示姿态脸图像左右两边具有相同点序的特征点列，

分别为所述第一网格网络中姿态脸图像左右两边对应的网格点的像素颜色。in,

respectively represent the feature point columns with the same point sequence on the left and right sides of the pose face image,

are the pixel colors of the grid points corresponding to the left and right sides of the pose face image in the first grid network, respectively.

The method for processing face images based on grid deformation optimization, wherein the first grid network that has undergone the initial optimization is performed according to the first optimization function, the second optimization function and the third optimization function. Re-optimization includes:

A first mesh network that minimizes the function values of the first optimization function, the second optimization function, and the third optimization function is obtained as an optimization result.

The described face image processing method based on grid deformation optimization, wherein, converting the posture face image into a target frontal face image according to the optimized first grid network includes:

Converting the pose face image into an intermediate frontal face image according to the optimized first grid network;

Correcting the middle frontal face image according to the fourth preset formula, and obtaining the target frontal face image;

Wherein, the fourth preset formula is:

Wherein, OL _p is the brightness of the pixel p in the occluded area of the intermediate frontal face image, OL _q is the brightness of the pixel q in the neighborhood of the p pixel, NL _p and NL _q are the same as the occluded area, respectively. The brightness of the pixel points corresponding to the pixel point p and the pixel point q in the corresponding non-occluded area, N _p is the 8 neighborhoods of the pixel point p, CE _OL and CE _NL are the occluded area and the corresponding occluded area respectively. The magnitude of the illumination intensity change of the pixels in the bounding ring region in the non-occluded region.

A second aspect of the present invention provides a terminal, the terminal includes a processor and a storage medium communicatively connected to the processor, the storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the storage medium to execute the steps of implementing the mesh deformation optimization-based face image processing method described in any of the above.

According to a third aspect of the present invention, a storage medium is provided, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any one of the above The steps of a face image processing method based on mesh deformation optimization.

Compared with the prior art, the present invention provides a face image processing method, terminal and storage medium based on mesh deformation optimization and deformation optimization, and the face image processing method based on mesh deformation optimization is used for processing posture faces. image, obtain the second feature lattice of the predicted frontal face image according to the first feature lattice of the posture face image, and construct the first grid network of the posture face image and the second grid network of the predicted frontal face image, according to the The second grid network, the second feature lattice and the first feature lattice optimize the first grid network; according to the optimized first grid network, the posture face image is converted into For the target front face image, the present invention realizes the transformation of the gesture face image into the front face image, which enables the face recognition technology to recognize the gesture face image and improves the performance of the face recognition system.

Description of drawings

1 is a flowchart of an embodiment of a method for processing a face image based on mesh deformation optimization provided by the present invention;

2 is a schematic diagram of obtaining a first feature lattice of a pose face image in an embodiment of a face image processing method based on grid deformation optimization provided by the present invention;

3 is a schematic diagram of a face shape database in an embodiment of a face image processing method based on grid deformation optimization provided by the present invention;

4 is a schematic diagram of obtaining a second feature lattice in an embodiment of a face image processing method based on grid deformation optimization provided by the present invention;

FIG. 5 is a flowchart of sub-steps of step S300 of an embodiment of a face image processing method based on mesh deformation optimization provided by the present invention;

6 is a schematic diagram of generating an intermediate frontal face image in an embodiment of a method for optimizing face image processing based on mesh deformation provided by the present invention;

7 is a schematic diagram of generating a target frontal face image in an embodiment of a method for optimizing face image processing based on mesh deformation provided by the present invention;

FIG. 8 is a schematic diagram of the principle of an embodiment of a terminal provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and effects of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Example 1

The face image processing method based on grid deformation optimization provided by the present invention may be applied in a terminal, and the terminal can process the pose face image through the face image processing method based on grid deformation optimization provided by the present invention, Convert the pose face image to the front face image. As shown in FIG. 1, an embodiment of the method for processing face images based on mesh deformation optimization includes the steps:

S100. Acquire a first feature lattice of the pose face image and a second feature lattice of the predicted front face image corresponding to the pose face image.

In this embodiment, the frontal face image represents the face image facing straight ahead, and the posture face image is a face image that is not a frontal face image. When the posture face image needs to be converted into a frontal face image, the posture face image is obtained first. The first feature lattice of the image, the first feature lattice includes each first feature point, that is, the first feature lattice is obtained by extracting multiple feature points in the posture face image (as shown in Figure 2) , the second feature lattice includes each second feature point, the number of the first feature point in the first feature lattice is the same as the number of the second feature point in the second feature lattice In a possible implementation manner, the number of feature points in the first feature lattice and the second feature lattice may both be 68, and the first feature point may be in the pose face Feature points are extracted from the image. Obtaining the second feature lattice of the predicted frontal face image corresponding to the posture face includes:

S110. Build a face shape database, and obtain the feature vector of the face shape database.

The predicted frontal face image is a frontal face image estimated according to the posture face image, that is, the predicted frontal face image is a virtual object. In this embodiment, the posture is obtained by a PCA (Principal Component Analysis) method. The second feature lattice of the predicted frontal face image corresponding to the face image, specifically, a face shape database is first constructed, as shown in Figure 3, and a plurality of face images are stored in the face shape database, For each face image, n feature points are extracted, and the feature lattice corresponding to each face image (the feature lattice corresponding to the face image can also be called the shape of the face) can be represented by a vector O, O=( x ₁ , y ₁ ... x _n , y _n ) ^T , where (x _n , y _n ) represents the nth feature point. S120. Acquire the second feature lattice according to a first preset formula.

Specifically, the first preset formula is:

为所述人脸形状数据库的平均形状，O为所述姿态脸图像的人脸形状，n₀为常数，而n₀-1表示去掉的特征向量的个数,也就是说，在获取所述第二特征点阵时，去掉所述人脸形状数据库的前n₀-1个特征向量，n₀可以设置为2,3等。根据上述公式，可以获取到所述姿态脸图像对应的预测正脸图像的形状(所述第二特征点阵)，如图4所示。Wherein, Q ₀ is the vector representation of the second feature lattice, E _i represents the ith feature vector of the face shape database,

is the average shape of the face shape database, O is the face shape of the posture face image, n ₀ is a constant, and n ₀ -1 represents the number of removed feature vectors, that is, when obtaining the In the second feature lattice, the first n ₀ -1 feature vectors of the face shape database are removed, and n ₀ can be set to 2, 3, etc. According to the above formula, the shape of the predicted frontal face image corresponding to the posture face image (the second feature lattice) can be obtained, as shown in FIG. 4 .

Please refer to FIG. 1 again, the face image processing method based on mesh deformation optimization further includes the steps:

S200. Build a first grid network corresponding to the pose face image and a second grid network corresponding to the predicted front face image, respectively.

In a possible implementation manner, in order to obtain more feature points and improve the accuracy of image processing, the first grid network corresponding to the pose face image and the second grid network corresponding to the predicted face image are respectively constructed. The mesh network also includes steps before:

The number of the first feature points in the first feature lattice and the number of the second feature points in the second feature lattice are expanded.

来实现，其中，RS、Q′分别为扩充后的所述第一特征点阵以及所述第二特征点阵，Q₀、RS₀分别为扩充前的所述第一特征点阵以及所述第二特征点阵,b、T、C分别为预设的尺度系数、变换矩阵和平移矩阵。b、T、C可以通过样本图像预先求解计算得到，具体地，在本实施例中，对所述第一特征点阵以及所述第二特征点阵中的特征点数量是增加人脸额头部分的特征点，扩充后的所述第一特征点阵以及所述第二特征点阵中的特征点数量可以为79个。The number of feature points in the first feature lattice and the second feature lattice can be expanded through the formula:

to realize, wherein RS and Q′ are respectively the expanded first feature lattice and the second feature lattice, Q ₀ and RS ₀ are respectively the first feature lattice before expansion and the second feature lattice The second feature lattice, b, T, and C are preset scale coefficients, transformation matrices and translation matrices, respectively. b, T, and C can be obtained by pre-solving and calculating the sample image. Specifically, in this embodiment, the number of feature points in the first feature lattice and the second feature lattice is increased by adding the forehead part of the face. The number of feature points in the expanded first feature lattice and the second feature lattice may be 79.

The step of constructing the first grid network corresponding to the posture face image and the second grid network corresponding to the predicted frontal face image respectively includes:

respectively constructing the first mesh network and the second mesh network according to a second preset formula;

The second preset formula is:

P _i+1,j +P _i-1,j +P _i,j+1 +P _i,j-1 -4P _i,j =0

_i =0,...,Nu; j=0,..., _Nv

Among them, P _i,j is a grid point in the i-th row and j-th column in the grid, N _u +1 is the number of rows of the grid, and N _v +1 is the number of columns of the grid.

The boundary conditions of the mesh are:

The left boundary line, right boundary line, upper boundary line and lower boundary line of the grid are respectively: P _0,j , P _1,j , P _i,0 and P _i,1 .

The first mesh network and the second mesh network are respectively constructed according to the above formulas. It is not difficult to see that the initial shapes of the first mesh network and the second mesh network are the same. In the processing step, the second mesh network remains unchanged, and the first mesh network is optimized and adjusted.

S300. Optimize the first grid network according to the second grid network, the second feature lattice, and the first feature lattice.

After the first mesh network and the second mesh network are constructed, the first mesh network is optimized, and the optimization of the first mesh network is to adjust the The position of each grid point, the optimization purpose is to make the first relative distance of each first feature point relative to the grid point in the first grid network and the each second feature point relative to the second grid network. The second relative distances of the grid points in the grid are consistent, so as to achieve the purpose of processing the pose face image into a frontal face image according to the first grid network.

Specifically, as shown in FIG. 5 , the positions of the first feature points and the first grid network are optimized according to the second grid network and the second feature lattice, and the The processing of the pose face image into a frontal face image includes:

S310. Perform initial optimization on the first mesh network according to a third preset formula;

The third preset formula is:

Wherein, P _i,j , P′ _i,j are the grid points of the first grid network and the second grid network, respectively, Q _t , Q′ _t are the grid points from the first grid network The t-th first feature point in the grid starting from the grid point P _i,j and the t-th second feature point in the grid starting from the grid point P′ _i,j in the second grid network feature points, iteratively optimizes the positions of grid points in the first grid network by using the third preset formula.

It is worth noting that Q _t may not be in the grid starting from P _i,j at the beginning, and through the third preset formula, the corresponding point Q _t will gradually approach the grid point P after a number of iterations. _i,j .

S320. Re-optimize the first grid network that has performed the initial optimization according to the first optimization function, the second optimization function, and the third optimization function;

After performing the step S310, the grid point positions in the first grid network are preliminarily optimized, and in the step S320, the first grid network is further optimized.

In the step S320, the image deformation optimization is automatically performed using the structure and texture similarity in the intermediate domain. Specifically, the first optimization function, the second optimization function and the third optimization function are based on smoothness, translation Transgender and the left-right symmetry of the human face.

The first optimization function is used to constrain the smoothness of the frontal face image obtained by converting the posture face image according to the optimized first grid network. The better the smoothness of the image, the first optimization function is:

E _TPS (z(P _i,j ))=(zx″ _u,u ) ² +2(zx″ _u,v ) ² +(zx″ _v,v ) ² +(zy″ _u,u ) ² +2 (zy″ _u,v ) ² +(zy″ _v,v ) ²

Among them, z(P _i,j )=(zx,zy), represents the offset of the P i, _j grid point, zx is the offset of the Pi _,j grid point in the u direction, and zy is P The offset of _i,j grid points in the v direction, zx″ _u,v represents the second-order directional partial derivative of zx with respect to the u direction and the v direction, that is, in this embodiment, the smoothness is expressed as the face network The sum of thin plate splines (TPS) in the x and y directions.

The second optimization function is used to constrain the translation invariance of the frontal face image obtained by converting the pose face image according to the optimized first grid network, and the smaller the function value of the second optimization function, The translation invariance of the frontal face image is better, and the second optimization function is:

Wherein, z(Q _t )=Q′ _t −Q _t , which represents the translation vector between Q _t and Q′ _t .

The third optimization function is used to constrain the left-right symmetry of the frontal face image obtained by converting the posture face image according to the first grid network. The smaller the function value of the third optimization function, the left-right symmetry of the frontal face image. Specifically, in this embodiment, the left-right symmetry includes shape symmetry and texture symmetry, and the third optimization function is:

分别表示姿态脸图像左右两边具有相同点序的特征点列，

分别为所述第一网格网络中姿态脸图像左右两边对应的网格点的像素颜色。根据公式

可以将约束形状对称性的函数中的特征点转化为网格点，使得网格点作为函数L_SymShape中的变量，实现通过函数L_SymShape对所述第一网格网络进行优化。Among them, L _SymShape is a function that constrains shape symmetry, and L _SymTex is a function that constrains texture symmetry

respectively represent the feature point columns with the same point sequence on the left and right sides of the pose face image,

are the pixel colors of the grid points corresponding to the left and right sides of the pose face image in the first grid network, respectively. According to the formula

The feature points in the function constraining the shape symmetry can be converted into grid points, so that the grid points are used as variables in the function L _SymShape to realize the optimization of the first grid network by the function L _SymShape .

The re-optimization of the first grid network after the initial optimization is performed by solving the first optimization function, the second optimization function and the third optimization function. The first optimization function, the second optimization function and the third optimization function to re-optimize the first grid network after the initial optimization includes:

A first mesh network that minimizes the function values of the first optimization function, the second optimization function, and the third optimization function is obtained as an optimization result.

The relationship between the first optimization function, the second optimization function and the third optimization function and the smoothness, translation invariance and left-right symmetry of the image has been described above. Therefore, by making the first optimization function , the second optimization function and the third optimization function sum to the minimum value to optimize the first grid network (adjust the position and pixel position of each grid point in the first grid network value), in this way, the image quality generated according to the optimized first mesh network is higher.

Please refer to FIG. 1 again, the face image processing method based on mesh deformation optimization further includes the steps:

S400. Convert the pose face image into a target front face image according to the optimized first grid network.

Specifically, converting the pose face image into a target frontal face image according to the optimized first grid network includes:

S410, according to the optimized first grid network, the posture face image is converted into a middle frontal face image;

S420. Correct the middle frontal face image according to a fourth preset formula to acquire the target frontal face image.

An image corresponding to the first grid network may be generated according to the optimized first grid network, which is a prior art and will not be repeated here. In a possible implementation manner, the intermediate frontal face image generated in the step S410 is directly used as the target frontal face image, that is, as the processing result of the posture face image. However, for a face image with a relatively large tilt posture, the posture face will contain a large occluded part, and some areas of the middle frontal face image generated according to the posture face may have texture loss (as shown in the figure). 6), the area where the texture is lost is called the occluded area. In this embodiment, the fourth preset formula is used to process the intermediate frontal face image to generate the final target frontal face image.

The middle frontal face image is processed using a Poisson-based inpainting method, and the conventional filling algorithm formula is:

Among them, OL _p is the brightness of pixel p in the occluded area in the image, OL _q is the brightness of pixel q in the neighborhood of p pixel point, NL _p , NL _q are the non-occluded area corresponding to the occluded area and The brightness of the pixel points corresponding to the pixel point p and the pixel point q, N _p is the 4 neighborhoods of the pixel point p, ie |N _p |=4.

In this embodiment, after the above formula is improved, the middle face image is processed, and the fourth preset formula is:

Wherein, OL _p is the brightness of the pixel p in the occluded area of the intermediate frontal face image, OL _q is the brightness of the pixel q in the neighborhood of the p pixel, NL _p and NL _q are the same as the occluded area, respectively. The brightness of the pixel points corresponding to the pixel point p and the pixel point q in the corresponding non-occluded area, N _p is the 8 neighborhoods of the pixel point p, CE _OL and CE _NL are the occluded area and the corresponding occluded area respectively. The magnitude of the illumination intensity change of the pixels in the bounding ring region in the non-occluded region.

It is not difficult to see from the above description that in this embodiment, the number of adjacent pixels is changed from 4 to 8, the number of equations corresponding to each pixel is changed from 1 to 8, and the brightness difference is obtained The ratio coefficient of , improves the detail information obtained during filling and improves the authenticity of the filling area. The illumination intensity of the abnormal area in the intermediate face image is obtained by using the fourth preset formula to obtain the target frontal face image, as shown in FIG. 7 .

To sum up, this embodiment provides a face image processing method based on grid deformation optimization. The first feature lattice obtains the second feature lattice for predicting the frontal face image, and constructs the first grid network for the posture face image and the second grid network for predicting the frontal face image. The second feature lattice and the first feature lattice optimize the first grid network; according to the optimized first grid network, the posture face image is converted into a target face image, the present invention realizes In order to convert the gesture face image into a front face image, the face recognition technology can recognize the gesture face image and improve the performance of the face recognition system.

It should be understood that although the various steps in the flowcharts given in the accompanying drawings of the present invention are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence is also not necessarily sequential, but may be performed alternately or alternately with other steps or sub-steps of other steps or at least a portion of a phase.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided by the present invention may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Embodiment 2

Based on the above embodiments, the present invention also provides a terminal correspondingly. As shown in FIG. 8 , the terminal includes a processor 10 and a memory 20 . It can be understood that FIG. 8 only shows some components of the terminal, but it should be understood that it is not required to implement all the shown components, and more or less components may be implemented instead.

In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD card) equipped on the terminal. ) card, flash card (Flash Card) and so on. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used for storing application software and various types of data installed in the terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a face image processing program 30 optimized based on mesh deformation, and the face image processing program 30 optimized based on mesh deformation can be executed by the processor 10, thereby realizing the A face image processing method based on mesh deformation optimization.

In some embodiments, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a microprocessor or other chips, for running the program codes or processing data stored in the memory 20, such as executing all Describe the face image processing method based on mesh deformation optimization, etc.

In one embodiment, when the processor 10 executes the face image processing program 30 optimized based on mesh deformation in the memory 20, the following steps are implemented:

Obtain the first feature lattice of the pose face image and the second feature lattice of the predicted frontal face image corresponding to the pose face image, wherein the first feature lattice includes each first feature point, and the second feature The lattice includes each second feature point;

respectively constructing a first grid network corresponding to the first feature lattice and a second grid network corresponding to the second feature lattice;

Optimizing the first grid network according to the second grid network, the second feature lattice and the first feature lattice;

The pose face image is converted into a target front face image according to the optimized first mesh network.

Wherein, obtaining the second feature lattice of the predicted frontal face image corresponding to the posture face image includes:

constructing a face shape database, and obtaining the feature vector of the face shape database;

The second feature lattice is obtained according to the first preset formula,

Wherein, the first preset formula is:

为所述人脸形状数据库的平均形状，O为所述姿态脸图像的人脸形状，n₀为常数，而n₀-1表示去掉的特征向量的个数。Wherein, Q ₀ is the vector representation of the second feature lattice, E _i represents the ith feature vector of the face shape database,

is the average shape of the face shape database, O is the face shape of the pose face image, n ₀ is a constant, and n ₀ -1 represents the number of removed feature vectors.

Wherein, before respectively constructing the first grid network corresponding to the first feature lattice and the second grid network corresponding to the second feature lattice, the steps include:

The number of the first feature points in the first feature lattice and the number of the second feature points in the second feature lattice are expanded.

Wherein, constructing the first grid network corresponding to the posture face image and the second grid network corresponding to the predicted frontal face image respectively includes:

respectively constructing the first mesh network and the second mesh network according to a second preset formula;

Wherein, the second preset formula is:

P _i+1,j +P _i-1,j +P _i,j+1 +P _i,j-1 -4P _i,j =0

_i =0,...,Nu; j=0,..., _Nv

Among them, P _i,j is a grid point in the i-th row and j-th column in the grid, N _u +1 is the number of rows of the grid, and N _v +1 is the number of columns of the grid.

Wherein, the optimizing the positions of the first feature points and the first grid network according to the second grid network, the second feature lattice and the first feature lattice includes:

Perform initial optimization on the first mesh network according to a third preset formula;

Re-optimize the first grid network on which the initial optimization has been performed according to the first optimization function, the second optimization function and the third optimization function;

Wherein, the third preset formula is:

Wherein, P _i,j , P′ _i,j are the grid points of the first grid network and the second grid network, respectively, Q _t , Q′ _t are the grid points from the first grid network The t-th first feature point in the grid starting from the grid point P _i,j and the t-th second feature point in the grid starting from the grid point P′ _i,j in the second grid network Feature points;

The first optimization function, the second optimization function and the third optimization function are respectively constructed based on smoothness, translation invariance and left-right symmetry of the human face.

Wherein, the first optimization function is:

E _TPS (z(P _i,j ))=(zx″ _u,u ) ² +2(zx″ _u,v ) ² +(zx″ _v,v ) ² +(zy″ _u,u ) ² +2 (zy″ _u,v ) ² +(zy″ _v,v ) ²

Among them, z(P _i,j )=(zx,zy), represents the offset of the P i, _j grid point, zx is the offset of the Pi _,j grid point in the u direction, and zy is P The offset of _{i, j} grid points in the v direction, zx″ _{u, v} represents the second-order partial derivative of zx with respect to the u and v directions;

The second optimization function is:

Wherein, z(Q _t )=Q' _t -Q _t , which represents the translation vector between Q _t and Q'_t;

The third optimization function is:

分别表示姿态脸图像左右两边具有相同点序的特征点列，

分别为所述第一网格网络中姿态脸图像左右两边对应的网格点的像素颜色。in,

respectively represent the feature point columns with the same point sequence on the left and right sides of the pose face image,

are the pixel colors of the grid points corresponding to the left and right sides of the pose face image in the first grid network, respectively.

Wherein, the re-optimization of the first grid network after the initial optimization according to the first optimization function, the second optimization function and the third optimization function includes:

A first mesh network that minimizes the function values of the first optimization function, the second optimization function, and the third optimization function is obtained as an optimization result.

Wherein, converting the pose face image into a target frontal face image according to the optimized first grid network includes:

Converting the pose face image into an intermediate frontal face image according to the optimized first grid network;

Correcting the middle frontal face image according to the fourth preset formula, and obtaining the target frontal face image;

Wherein, the fourth preset formula is:

Wherein, OL _p is the brightness of the pixel p in the occluded area of the intermediate frontal face image, OL _q is the brightness of the pixel q in the neighborhood of the p pixel, NL _p and NL _q are the same as the occluded area, respectively. The brightness of the pixel points corresponding to the pixel point p and the pixel point q in the corresponding non-occluded area, N _p is the 8 neighborhoods of the pixel point p, CE _OL and CE _NL are the occluded area and the corresponding occluded area respectively. The magnitude of the illumination intensity change of the pixels in the bounding ring region in the non-occluded region.

Embodiment 3

The present invention also provides a storage medium, in which one or more programs are stored, and the one or more programs can be executed by one or more processors, so as to realize the above-mentioned mesh-based deformation optimization of the human face The steps of an image processing method.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A face image processing method based on mesh deformation optimization is characterized by comprising the following steps:

acquiring a first feature lattice of an attitude face image and a second feature lattice of a predicted front face image corresponding to the attitude face image, wherein the first feature lattice comprises each first feature point, and the second feature lattice comprises each second feature point;

respectively constructing a first grid network corresponding to the first characteristic lattice and a second grid network corresponding to the second characteristic lattice;

optimizing the first mesh network according to the second mesh network, the second feature lattice and the first feature lattice;

and converting the attitude face image into a target front face image according to the optimized first grid network.

2. The method for processing the human face image based on the mesh deformation optimization according to claim 1, wherein the obtaining of the second feature lattice of the predicted front face image corresponding to the pose face image comprises:

constructing a face shape database, and acquiring a feature vector of the face shape database;

obtaining the second characteristic lattice according to a first preset formula,

wherein the first preset formula is as follows:

wherein ,Q₀For the vector representation of said second feature lattice, E_iRepresenting the ith feature vector of the face shape database,

is the average shape of the face shape database, O is the face shape of the posed face image, n₀Is a constant number, n₀-1 represents the number of removed feature vectors.

3. The method for processing a face image based on mesh deformation optimization according to claim 1, wherein before the respectively constructing a first mesh network corresponding to the first feature lattice and a second mesh network corresponding to the second feature lattice, the method comprises:

and expanding the number of the first characteristic points in the first characteristic lattice and the number of the second characteristic points in the second characteristic lattice.

4. The method for processing a human face image based on mesh deformation optimization according to claim 1, wherein the respectively constructing a first mesh network corresponding to the pose face image and a second mesh network corresponding to the predicted front face image comprises:

respectively constructing the first mesh network and the second mesh network according to a second preset formula;

wherein the second preset formula is as follows:

P_i+1,j+P_i-1,j+P_i,j+1+P_i,j-1-4P_i,j＝0

i＝0,…,N_u；j＝0,…,N_v

wherein ,P_i,jFor a grid point in the grid network, which is located at the ith row and the jth column, N_u+1 is the number of rows in the mesh network, N_v+1 is the number of columns in the mesh network.

5. The method for processing a facial image based on mesh deformation optimization according to claim 1, wherein the optimizing the positions of the first feature points and the first mesh network according to the second mesh network, the second feature point lattice and the first feature point lattice comprises:

performing primary optimization on the first grid network according to a third preset formula;

re-optimizing the first mesh network subjected to the primary optimization according to a first optimization function, a second optimization function and a third optimization function;

wherein the third preset formula is as follows:

wherein ,P_i,j,P′_i,jMesh points, Q, of the first mesh network and the second mesh network, respectively_t，Q′_tRespectively P from the first mesh network_i,jT-th first feature point in mesh starting at mesh point and P 'from the second mesh network'_i,jThe t-th second feature point in the grid from the grid point;

the first optimization function, the second optimization function and the third optimization function are respectively constructed based on smoothness, translation invariance and face bilateral symmetry.

6. The method for processing a facial image based on mesh deformation optimization according to claim 5, wherein the first optimization function is:

E_TPS(z(P_i,j))＝(zx″_u,u)²+2(zx″_u,v)²+(zx″_v,v)²+(zy″_u,u)²+2(zy″_u,v)²+(zy″_v,v)²

wherein ,z(P_i,j) (zx, zy) represents P_i,jOffset of grid points, zx being P_i,jThe grid points are offset in the u direction by an amount of P_i,jOffset of grid points in the v direction, zx ″_u,vRepresents the second directional partial derivatives of zx with respect to the u and v directions;

the second optimization function is:

E_TI(z(P_i,j))＝(1-α′-β′)||z(P_i,j)-z(Q_t)||²+α′||z(P_i,j+1)-z(Q_t)||²+β′||z(P_i+1,j)-z(Q_t)||²

wherein ,z(Q_t)＝Q″_t-Q_tDenotes Q_t，Q″_tA translation vector therebetween;

the third optimization function is:

wherein ,

respectively represent characteristic point columns with the same point sequence on the left side and the right side of the pose face image,

and the pixel colors of grid points corresponding to the left side and the right side of the attitude face image in the first grid network are respectively.

7. The method of claim 6, wherein the re-optimizing the first mesh network subjected to the initial optimization according to a first optimization function, a second optimization function and a third optimization function comprises:

and acquiring a first mesh network which enables the function value of the first optimization function, the second optimization function and the third optimization function to be minimum as an optimization result.

8. The mesh deformation optimization-based facial image processing method according to claim 1, wherein the converting the pose face image into the target front face image according to the optimized first mesh network comprises:

converting the attitude face image into a middle front face image according to the optimized first grid network;

correcting the middle front face image according to a fourth preset formula to obtain the target front face image;

wherein the fourth preset formula is:

wherein ,OL_pIs the brightness, OL, of the pixel points p in the shielded region of the intermediate front face image_qIs the brightness, NL, of pixel q in the neighborhood of p pixels_p、NL_qBrightness, N, of pixel points corresponding to pixel point p and pixel point q in a non-shielding region corresponding to the shielded region, respectively_p8 neighbourhood of pixel point p, CE_OL、CE_NLThe variation range of the illumination intensity of the pixel points in the boundary ring region in the sheltered region and the non-sheltered region corresponding to the sheltered region are respectively.

9. A terminal, characterized in that the terminal comprises: a processor, and a storage medium communicatively connected to the processor, the storage medium being adapted to store a plurality of instructions, the processor being adapted to call the instructions in the storage medium to execute the steps of implementing the mesh deformation optimization-based face image processing method according to any one of the preceding claims 1 to 8.

10. A storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the steps of the mesh deformation optimization-based face image processing method according to any one of claims 1 to 8.

Images