CN117690163A - A method and system for 3D human skeleton key point data enhancement - Google Patents

Abstract

The invention discloses a method and a system for enhancing 3D human skeleton key point data, wherein the method comprises the following steps: acquiring 2D bone key point data; determining a 2D bone key index of the missing limb edge connection and the number of missing bone keys; performing frame screening and smoothing on the 2D bone key point data; constructing a gesture priori constraint and data enhancement algorithm of the missing limb; and iterating the SMPLify-X model through the gesture priori constraint and data enhancement algorithm to obtain an optimal 3D skeleton SMPL-X model, and obtaining 3D skeleton key point data and a data annotation file thereof. The system comprises: the system comprises a data acquisition module, a data comparison analysis module, a data smoothing module, an algorithm construction module, an optimal model generation module and a labeling data acquisition module. By using the method, the consistency of the characteristic distribution of the key point data of the body of the missing limb and the key point data of the whole skeleton of the human body can be maintained, and the robustness and generalization capability of the algorithm are enhanced. The invention can be widely applied to the technical field of computer vision.

Description

A method and system for 3D human skeleton key point data enhancement

Technical field

The invention relates to the field of computer vision technology, and in particular to a 3D human skeleton key point data enhancement method and system.

Background technique

Human skeleton key point data is the necessary basis for some computer vision tasks, such as action classification, behavior recognition, and human body weight recognition, etc. At present, human skeleton key point data is mainly detected and obtained from complete human body images or video sequences (such as the NTU-RGB+D and NTU-RGB+D 120 data sets that are currently widely studied). Existing computer vision algorithms use complete human skeleton key point data as input and cannot adapt well to human skeleton key point data with missing limbs; for images with missing body limbs, existing human skeleton key point detection algorithms cannot Effective completion and data enhancement of key points of missing limb parts.

Contents of the invention

In order to solve the above technical problems, the goal of the present invention is to provide a 3D human skeleton key point data enhancement method and system, which can maintain the consistency of the feature distribution of missing limb human body key point data and complete human skeleton key point data, and enhance the robustness of the algorithm. stickiness and improve the generalization ability of the algorithm.

The first technical solution adopted by the present invention is: a 3D human skeleton key point data enhancement method, which includes the following steps:

Obtain the 2D bone key point data of the character in the missing limb video frame image;

Determine the 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

Construct posture prior constraints and data enhancement algorithms for missing limbs based on the number of missing bone key points, the 2D bone key point index of edge connections of the missing limb, and video behavior categories;

Taking the 2D skeleton key point data of the missing limb video frame image as input, the SMPLify-X model is iterated through the posture prior constraints of the missing limb and the data enhancement algorithm to obtain the optimal 3D skeleton SMPL-X model.

Further, it also includes obtaining the 3D skeleton key point data of the optimal 3D skeleton SMPL-X model and generating the corresponding data annotation file.

Further, after the step of determining the 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data, it also includes: Perform frame screening and smoothing processing on the 2D skeletal key point data of the missing limb video frame image to obtain smoothed 2D skeletal key point data.

Further, the step of performing frame screening and smoothing processing on the 2D skeletal key point data of the missing limb video frame image to obtain the smoothed 2D skeletal key point data specifically includes:

Perform polynomial regression on the same 2D bone key points in all frames of the missing limb video, and calculate the deviation between the regression value of the 2D bone key point and the initial value of the 2D bone key point;

Determine abnormal bone points based on the comparison results between the deviation and the minimum threshold and maximum threshold, and update the values of the 2D bone key points to obtain the first 2D bone key point data;

The first 2D bone key point data is filtered based on the comparison result between the number of abnormal bone points and the maximum number of abnormal bone points in the same frame image to obtain smoothed 2D bone key point data.

Through this optimization step, some abnormal bone point data are filtered out, so that the smoothed 2D bone key point data can maintain the complete characteristics of the missing limb human body key points.

Further, the step of constructing the posture prior constraints and data enhancement algorithm of the missing limb based on the number of the missing limb key points, the 2D skeletal key point index of the missing limb edge connection and the video behavior category specifically includes:

Obtain the complete human posture parameters, and perform human posture parameter mapping based on the 2D bone key point index connected to the missing limb edge, and obtain the human posture parameters corresponding to the missing bone key points;

Calculate the human posture parameters corresponding to the 2D bone key point data of the missing limb video frame image based on the complete human posture parameters and the human posture parameters corresponding to the missing bone key points;

Constrain the human posture parameters corresponding to the missing bone key points according to the video behavior category, the number of the missing bone key points and the 2D bone key point index connected to the missing limb edge, and obtain the posture prior constraint parameters of the missing bone key points. ;

Constraint terms for missing limb prior knowledge are constructed based on the posture prior constraint parameters of missing bone key points;

Construct the constraints of the whole body posture based on the complete human posture parameters;

Construct body part posture constraints based on the human posture parameters corresponding to the 2D skeletal key point data of the missing limb video frame image;

Combining the constraints of the missing limb's prior knowledge, the constraints of the whole body posture and the posture constraints of the body parts, the posture prior constraints of the missing limb and the data enhancement algorithm are obtained.

Through this optimization step, pose prior constraint completion and data enhancement are performed on human skeleton points with missing limbs, and the consistency of feature distribution of human key point data of missing limbs and complete human skeleton key point data is maintained, which can well enhance human body-based key point data. The robustness of the computer vision algorithm for bone key points improves the generalization performance of its algorithm.

Furthermore, the expression of the posture prior constraint and data enhancement algorithm of the missing limb is as follows:

θ' _{m, t} = Z (θ _{m, t} , N _{m, t} , I _{e, t} , C)

θ _d,t =θ _b,t -θ _m,t

Among them, E(β _t , θ _b,t , θ _d,t , θ′ _m,t , K _t , J′ _est,t ) represents the objective optimization function of the posture prior constraint and data augmentation algorithm based on the missing limb, Constraint items representing the whole body posture,/> Represents body part posture constraints,/> Constraints representing missing prior knowledge of limbs,/> The weight value of the constraint item representing the whole body posture,/> Represents the weight value of the body part posture constraint item,/> Represents the weight value of the constraint item with missing limb prior knowledge, β _t represents the body shape parameter corresponding to the t-th video frame image, θ _{b, t} represents the complete human posture parameter corresponding to the t-th video frame image, K _t represents the t-th video frame image The camera parameters corresponding to the frame video frame image, J′ _{est, t} represents the smoothed 2D skeleton key point data, θ _{d, t} represents the human posture parameter corresponding to the 2D skeleton key point data of the missing limb video frame image, θ' _{m, t} represents the posture prior constraint parameter of the missing bone key point corresponding to the t-th video frame image, θ _{m, t} represents the human posture parameter corresponding to the missing bone key point, N _{m, t} represents the total bones of the missing limb in the t-th frame The number of key points, I _e,t represents the set of index values of the edge connection bone points of all missing limb bones in the tth frame, and C represents the behavior category of the video sequence.

Furthermore, the 2D skeleton key point data of the missing limb video frame image is used as input, and the SMPLify-X model is iterated through the posture prior constraints of the missing limb and the data enhancement algorithm to obtain the optimal 3D skeleton SMPL-X model. It specifically includes:

Based on the SMPLify-X model and the pose prior constraints of the missing limb and the data enhancement algorithm, 3D human pose estimation is performed on the 2D skeletal key point data of the missing limb video frame image, and the SMPL-X model of the 2D skeletal key point data is obtained;

The SMPL-X model based on 2D bone key point data obtains 3D bone key point data, and maps the 3D bone key point data to 2D bone key point data on the image plane through camera parameters;

Match the 2D bone key point data on the image plane with the 2D bone key point data of the missing limb video frame image to obtain the matching result;

Based on the matching results, the pose prior constraints of the missing limbs and the weights of the data enhancement algorithm are optimized and iterated to obtain the optimal 3D skeleton SMPL-X model.

Through this optimization step, the feature fusion of the original bone points and the missing limb bone points is achieved. The matching degree between the original bone points and the missing limb bone points is higher, and the posture of the missing limb part and the overall posture of the three-dimensional human body model are more accurate.

The second technical solution adopted by the present invention is: a 3D human skeleton key point data enhancement system, including:

The data acquisition module is used to obtain the 2D skeletal key point data of the character in the video frame image of the missing limb;

The data comparison analysis module determines the 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

The data smoothing module is used to perform frame screening and smoothing processing on the 2D skeletal key point data of missing limb video frame images to obtain smoothed 2D skeletal key point data;

An algorithm building module that constructs a priori constraints on the posture of the missing limb and a data enhancement algorithm based on the number of missing bone key points, the 2D bone key point index of the missing limb edge connection, and the video behavior category.

The optimal model generation module takes the smoothed 2D skeleton key point data as input, and iterates the SMPLify-X model through the posture prior constraints of the missing limbs and the data enhancement algorithm to obtain the optimal 3D skeleton SMPL-X model;

The annotation data acquisition module is used to obtain the 3D skeleton key point data of the optimal 3D skeleton SMPL-X model and generate the corresponding data annotation file.

The beneficial effects of the method and system of the present invention are: by constructing the posture prior constraint and data enhancement algorithm of the missing limb, the present invention performs posture prior constraint completion and data enhancement on the human skeleton points of the missing limb, and maintains the key points of the complete human skeleton. The consistency of data feature distribution with missing limb key point data helps computer vision task algorithms and models better understand and capture the key features of input data, and alleviates problems caused by data offset, allowing the algorithm to better adapt to the future. seen data, thereby enhancing the robustness of the algorithm and improving the generalization performance of the algorithm under different data distributions; on the premise of ensuring that the bone point features of the original missing limb video frame image are not lost, the original bone point is realized and feature fusion of missing limb bone points. The matching degree between the original bone points and the missing limb bone points is higher, and the posture of the missing limb part and the overall posture of the three-dimensional human body model are more accurate.

Description of the drawings

Figure 1 is a step flow chart of a 3D human skeleton key point data enhancement method of the present invention;

Figure 2 is a structural block diagram of a 3D human skeleton key point data enhancement system of the present invention;

Figure 3 is a schematic diagram of the overall step structure of a 3D human skeleton key point data enhancement method of the present invention;

Figure 4 is a schematic diagram of the complete 2D skeleton key point connection relationship of a 3D human skeleton key point data enhancement method of the present invention;

Figure 5 is a schematic diagram of the connection relationship between the 2D skeleton key points of the missing limb in a 3D human skeleton key point data enhancement method of the present invention;

Figure 6 is a schematic diagram comparing the effects of a 3D human skeleton key point data enhancement method of the present invention and the prior art. Figure 6(a) shows the original missing limb image (missing leg limb); Figure 6(b) shows the OpenPose detector Detected 2D skeleton key points and connection diagram; Figure 6(c) shows the SMPL-X human body 3D model obtained directly using the SMPLify-X model; Figure 6(d) shows the SMPL-X human body 3D model obtained through the present invention Figure; Figure 6(e) shows a schematic diagram of 3D bone key points extracted from the SMPL-X human body 3D model obtained directly using the SMPLify-X model; Figure 6(f) shows the SMPL-X human body 3D model obtained from the present invention. Schematic diagram of extracted 3D skeleton key points.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only set for the convenience of explanation. The order between the steps is not limited in any way. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art. sexual adjustment.

Referring to Figures 1 and 3, the present invention provides a 3D human skeleton key point data enhancement method, which method includes the following steps:

S1. Obtain the 2D skeletal key point data of the character in the missing limb video frame image;

Specifically, the missing limb video is processed into T missing limb video frame images according to the resolution of 300×256 and the frame rate of 30fps. The OpenPose algorithm is used to extract the 2D skeletal key point coordinate data of the character with missing limbs in each frame. The 2D skeletal key point coordinate data extracted at the tth frame is J _{est, t} . J _{est, t} is a 25×2 matrix. The rows represent 25 different joint points extracted by OpenPose. The node numbers are 0-24. The columns represent the x-coordinates and y-coordinates of the joint points. The complete 2D skeleton key point connection The relationship is shown in Figure 4.

S2. Determine the 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

Specifically, the 2D bone key point data of the missing limb video frame image is compared with the complete 2D bone key point data, and the 2D bone key point index at the connection between the existing bone and the missing limb is obtained through the edge connection index function, and its expression is as follows:

I _{e, t} = {n _t |n _t = Eage (J _{k, t} ); k = 1,..., N; t = 1,..., T}

Among them, Eage(J _k,t ) represents the edge connection index function, J _k,t represents the k-th bone key point detected in the t-th frame image, n _t represents the edge connection bone between the existing bone key point and the missing limb bone. The index value of the key point, I _e,t represents the set of index values of the edge connecting bone key points of all missing bones at frame t, N=25, represents the 25 different joint points of the human body detected by OpenPose, T represents the missing limb The total number of frames in the video sequence.

The number of missing bone key points is obtained through a statistical function, and its expression is as follows:

Nm _,t =CountBlank(Jk _,t )

J _{k, t} = J _{est, t} = {(x _{k, t} , y _{k, t} )|k=1,...,N; t=1,...,T}

Among them, N _{m, t} represents the number of missing bone key points in the t-th frame, CountBlank(J _{k, t} ) represents the statistical function of the number of missing bone key points in the bone key point sequence, x _{k, t} , y _{k, t} Respectively represent the x coordinate and y coordinate of the bone key point at the tth frame.

Refer to Figure 5, in which the black dots represent the key points of the bones detected by OpenPose, that is, the 2D key points of the bones in the missing limb video frame image; the gray points represent the key points of the bones corresponding to the missing limbs, and the three points 8, 9 and 12 represent the missing limbs. The edge of the bone key point corresponding to the limb and the detected bone key point connects the bone point. 8, 9 and 12 are the index values of the point.

S3. Perform frame screening and smoothing processing on the 2D skeletal key point data of the missing limb video frame image to obtain smoothed 2D skeletal key point data;

Specifically, the expressions of the frame filtering and smoothing processing are as follows:

J′ _{est, t} = Y (J _{est, t} , L, H, N _max )

Among them, J′ _{est, t} represents the smoothed 2D bone key point data, J _{est, t} represents the 2D bone key point data of the missing limb video frame image detected by the OpenPose algorithm, L represents the minimum threshold, H represents the maximum threshold, N _max represents the maximum number of abnormal bone points allowed to be discarded in each frame, and Y(J _{est, t} , L, H, N _max ) represents the frame filtering and smoothing function based on polynomial regression.

The specific operations of the frame filtering and smoothing functions of polynomial regression are as follows:

S3.1. Perform polynomial regression on the same 2D bone key points in all frames of the missing limb video to obtain the regression value, and then calculate the deviation between the regression value of the 2D bone key point in each frame and the initial value of the 2D bone key point;

S3.2. Determine abnormal bone points based on the comparison results between the deviation and the minimum threshold and maximum threshold, and update the values of the 2D bone key points to obtain the first 2D bone key point data;

Specifically, if the deviation is greater than the maximum threshold H, the bone point in this time frame is regarded as an abnormal bone point that is allowed to be discarded, and the abnormal bone point is discarded; if the deviation is between the maximum threshold H and the minimum threshold L, then the abnormal bone point is discarded. The bone points are abnormal bone points that are not allowed to be discarded. At this time, the regression value of the abnormal bone point needs to be replaced by the initial value to reduce the deviation; if the deviation is less than the minimum threshold L, the bone point is a normal bone point and should be retained. The initial value of the bone point finally obtains the first 2D bone key point data.

S3.3. Filter the first 2D bone key point data based on the comparison results between the number of abnormal bone points and the maximum number of abnormal bone points in the same frame image to obtain smoothed 2D bone key point data.

Specifically, if the statistics of the number of abnormal bone points allowed to be discarded in the same frame exceeds the maximum number of abnormal bone points allowed to be discarded in each frame N _max , then all the bone point data of the frame will be regarded as abnormal data and deleted. Obtain the smoothed 2D skeleton key point data J′ _est,t .

S4. Construct posture prior constraints and data enhancement algorithms for missing limbs based on the number of missing bone key points, the 2D bone key point index of the missing limb edge connection, and the video behavior category;

S4.1. Obtain the complete human posture parameters, and perform human posture parameter mapping based on the 2D bone key point index of the missing limb edge connection to obtain the human posture parameters corresponding to the missing bone key points;

Specifically, the expression of the human posture parameters corresponding to the missing bone key points is as follows:

θ _{m, t} = F (θ _{b, t} , I _{e, t} )

Among them, θ _{m, t} represents the human posture parameters corresponding to the missing bone key points, I _{e, t} represents the set of index values of the edge connecting bone points of the missing bones, θ _{b, t} represents the complete human posture in the SMPL-X human body model Parameter, F represents the mapping function, which uses the complete human posture parameters and the index of the edge connecting bone points of the missing bones to perform human posture parameter mapping, and obtains the human posture parameters corresponding to the missing bone key points.

S4.2. Calculate the human posture parameters corresponding to the 2D bone key point data of the missing limb video frame image based on the complete human posture parameters and the human posture parameters corresponding to the missing bone key points;

Specifically, the calculation expression of the human posture parameters corresponding to the 2D skeletal key point data of the missing limb video frame image is as follows:

θ _d,t =θ _b,t -θ _m,t

Among them, θ _{d, t} represents the human posture parameters corresponding to the 2D skeletal key point data of the missing limb video frame image detected by OpenPose.

S4.3. Constrain the human posture parameters corresponding to the missing bone key points according to the video behavior category, the number of the missing bone key points and the 2D bone key point index connected to the missing limb edge, and obtain the posture of the missing bone key point. A priori constraint parameters;

Specifically, the posture prior constraint parameters of the missing bone key points are expressed as follows:

θ' _{m, t} = Z (θ _{m, t} , N _{m, t} , I _{e, t} , C)

Among them, θ' _{m, t} represents the posture prior constraint parameters of the missing bone key points, θ _{m, t} represents the human posture parameters corresponding to the missing bone key points, and N _{m, t} represents the total bone key points of the missing limb in the tth frame. The quantity, I _e,t represents the set of index values of the edge connection bone points of all missing limb bones in the tth frame, C represents the behavior category of the video sequence, and Z represents the posture prior constraint function of the missing limb.

S4.4. Construct a constraint term for missing limb prior knowledge based on the posture prior constraint parameters of missing skeletal key points. The constraint term for missing limb prior knowledge is expressed as

S4.5. Construct the constraint term of the whole body posture based on the complete human body posture parameters. The constraint term of the whole body posture is expressed as

S4.6. Construct a body part posture constraint item based on the human posture parameters corresponding to the 2D skeletal key point data of the missing limb video frame image. The body part in the body part posture constraint item refers to the missing limb video frame in this embodiment. The body part in the image, the body part pose constraint is expressed as

S4.7. Combining the constraints of the missing limb’s prior knowledge, the constraints of the whole body posture and the posture constraints of the body parts, obtain the posture prior constraints of the missing limb and the data enhancement algorithm.

Specifically, the expression of the posture prior constraint and data enhancement algorithm of the missing limb is as follows:

Among them, E(β _t , θ _b,t , θ _d,t , θ′ _m,t , K _t , J′ _est,t ) represents the objective optimization function of the posture prior constraint and data augmentation algorithm based on the missing limb, Constraint items representing the whole body posture,/> Represents body part posture constraints,/> Constraints representing missing prior knowledge of limbs,/> The weight value of the constraint item representing the whole body posture,/> Represents the weight value of the body part posture constraint item,/> Represents the weight value of the constraint item with missing limb prior knowledge, β _t represents the body shape parameter corresponding to the t-th video frame image, θ _{b, t} represents the complete human posture parameter corresponding to the t-th video frame image, K _t represents the t-th video frame image The camera parameters corresponding to the frame video frame image, J′ _{est, t} represents the smoothed 2D skeleton key point data, θ _{d, t} represents the human posture parameter corresponding to the 2D skeleton key point data of the missing limb video frame image, θ' _{m, t} represents the posture prior constraint parameter of the missing bone key point corresponding to the t-th video frame image.

The whole body posture constraint can well avoid the problem of mutual penetration of various parts of the model during the optimization process; the body part posture constraint can well retain and further optimize the bones corresponding to existing body parts during the optimization process. point; the constraint term of the missing limb prior knowledge can well complete and data enhance the 3D skeletal points of the missing limb part based on the missing limb prior knowledge.

S5. Take the 2D skeleton key point data of the missing limb video frame image as input, and iterate the SMPLify-X model through the posture prior constraints of the missing limb and the data enhancement algorithm to obtain the optimal 3D skeleton SMPL-X model;

Specifically, after frame screening and smoothing processing of the 2D skeletal key point data of the missing limb video frame image, the input data is changed to the smoothed 2D skeletal key point data; based on the SMPLify-X model and the pose prior constraints of the missing limb and data enhancement algorithm to perform 3D human pose estimation on the smoothed 2D skeletal key point data, and obtain the SMPL-X model of the 2D skeletal key point data; the SMPL-X model based on the 2D skeletal key point data obtains the 3D skeletal key point data, and Map 3D bone key point data to 2D bone key point data on the image plane through camera parameters; match the 2D bone key point data on the image plane with the smoothed 2D bone key point data to obtain the matching result; based on the matching result The optimal 3D skeleton SMPL-X model was obtained by optimizing and iterating the pose prior constraints of the missing limbs and the weights of the data augmentation algorithm.

S6. Obtain the 3D skeleton key point data of the optimal 3D skeleton SMPL-X model and generate the corresponding data annotation file.

Referring to Figure 6, the effect of the method of the present invention is compared with the existing technology that does not use the method of the present invention. Figure 6(a) shows the original missing limb image (missing leg limb); Figure 6(b) shows the OpenPose detector detection The 2D skeleton key points and connecting bones are obtained; the SMPL-X human body 3D model obtained by combining the posture prior constraint and data enhancement algorithm of the missing limbs proposed by the present invention with the SMPLify-X model is shown in Figure 6(d). Using the posture prior constraints and data enhancement algorithms for missing limbs proposed by the present invention, the SMPL-X human body 3D model obtained directly using the SMPLify-X model is shown in Figure 6(c). Compare Figure 6(c) and Figure 6( d), it can be found that the SMPL-X human body 3D model obtained using the method of the present invention is more accurate in predicting human body posture and can better restore the posture of missing limbs in the image. The 3D bone key points extracted from the model shown in Figure 6(c) are shown in Figure 6(e), and the 3D bone key points extracted from the model shown in Figure 6(d) are shown in Figure 6(f). Comparing Figure 6(e) and Figure 6(f), it can be found that the posture prior constraints and data enhancement algorithm of the missing limb proposed by the present invention have been achieved in the completion and data enhancement of the 3D skeleton key point data of the missing limb. Excellent performance.

As shown in Figure 2, the present invention provides a 3D human skeleton key point data enhancement system, which includes:

The data acquisition module is used to obtain the 2D skeletal key point data of the character in the video frame image of the missing limb;

The data comparison analysis module determines the 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

The data smoothing module is used to perform frame screening and smoothing processing on the 2D skeletal key point data of missing limb video frame images to obtain smoothed 2D skeletal key point data;

An algorithm building module that constructs a priori constraints on the posture of the missing limb and a data enhancement algorithm based on the number of missing bone key points, the 2D bone key point index of the missing limb edge connection, and the video behavior category.

The optimal model generation module takes the smoothed 2D skeleton key point data as input, and iterates the SMPLify-X model through the posture prior constraints of the missing limbs and the data enhancement algorithm to obtain the optimal 3D skeleton SMPL-X model;

The annotation data acquisition module is used to obtain the 3D skeleton key point data of the optimal 3D skeleton SMPL-X model and generate the corresponding data annotation file.

The contents in the above method embodiments are applicable to this system embodiment. The specific functions implemented by this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The above is a detailed description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can also make various equivalent modifications or substitutions without violating the spirit of the present invention. , these equivalent modifications or substitutions are included in the scope defined by the claims of this application.

Claims (8)

1. The 3D human skeleton key point data enhancement method is characterized by comprising the following steps of:

acquiring 2D skeleton key point data of a person in a missing limb video frame image;

determining a 2D bone key point index of the missing limb edge connection and the number of missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

constructing a posture prior constraint and data enhancement algorithm of the missing limb based on the number of the missing bone key points, the 2D bone key point index of the missing limb edge connection and the video behavior category;

and taking 2D skeleton key point data of the missing limb video frame image as input, and iterating the SMPLify-X model through gesture priori constraint and data enhancement algorithm of the missing limb to obtain an optimal 3D skeleton SMPL-X model.

2. The method for enhancing 3D human skeletal key point data of claim 1, further comprising: and acquiring 3D skeleton key point data of the optimal 3D skeleton SMPL-X model, and generating a corresponding data annotation file.

3. The method for enhancing 3D human bone keypoint data according to claim 1, further comprising, after the step of determining the index of 2D bone keypoints and the number of missing bone keypoints of the missing limb edge connection based on the 2D bone keypoint data of the missing limb video frame image and the complete 2D bone keypoint data: and carrying out frame screening and smoothing treatment on the 2D skeleton key point data of the missing limb video frame image to obtain smoothed 2D skeleton key point data.

4. The method for enhancing 3D human skeleton key point data according to claim 3, wherein the step of performing frame filtering and smoothing on the 2D skeleton key point data of the missing limb video frame image to obtain smoothed 2D skeleton key point data specifically comprises:

polynomial regression is carried out on the same 2D bone key points in all frames of the missing limb video, and deviation between the regression value of the 2D bone key points and the initial value of the 2D bone key points is calculated;

determining abnormal bone points based on comparison results of the deviation, the minimum threshold value and the maximum threshold value, and updating values of the 2D bone key points to obtain first 2D bone key point data;

and screening the first 2D bone key point data according to the comparison result of the abnormal bone point number and the maximum abnormal bone point number of the same frame of image to obtain smoothed 2D bone key point data.

5. The method for enhancing 3D human skeletal keypoint data according to claim 1, wherein said step of constructing a gesture priori constraint and data enhancement algorithm for the missing limb based on the number of the missing skeletal keypoints, the 2D skeletal keypoint index of the missing limb edge connection, and the video behavior class specifically comprises:

acquiring complete human body posture parameters, and mapping the human body posture parameters according to the indexes of the 2D skeleton key points connected with the edges of the missing limbs to obtain the human body posture parameters corresponding to the key points of the missing bones;

calculating human body posture parameters corresponding to 2D bone key point data of the missing limb video frame image based on the complete human body posture parameters and the human body posture parameters corresponding to the missing bone key points;

constraining the human body posture parameters corresponding to the missing skeleton key points according to the video behavior category, the number of the missing skeleton key points and the 2D skeleton key point indexes connected with the edges of the missing limbs to obtain posture priori constraint parameters of the missing skeleton key points;

constructing constraint items lacking limb priori knowledge based on gesture priori constraint parameters of the key points of the lacking bones;

constructing constraint items of the whole body posture based on the complete human posture parameters;

constructing a body part posture constraint item based on the body posture parameters corresponding to the 2D skeleton key point data of the missing limb video frame image;

and combining the constraint item without the prior knowledge of the limb, the constraint item of the whole body posture and the constraint item of the posture of the body part to obtain the prior constraint of the posture of the limb and a data enhancement algorithm.

6. The method for enhancing 3D human skeletal key point data according to claim 1, wherein the posture prior constraint and data enhancement algorithm of the missing limb has the following expression:

θ′ _m,t ＝Z(θ _m，t ，N _m,t ，l _e,t ，C)

θ _d,t ＝θ _b，t -θ _m,t

wherein E (beta) _t ，θ _b，t ，θ _d，t ，θ′ _m，t ，K _t ，J′ _est，t ) Representing an objective optimization function based on the pose a priori constraints of the missing limb and the data enhancement algorithm,constraint item representing whole body posture->Representing body part posture constraints->Constraint item representing lack of a priori knowledge of the limb, +.>Weight value of constraint item representing whole body gesture, < ->Weight value representing body part pose constraints, < +.>Weight value, beta, of constraint item representing lack of prior knowledge of limb _t Representing the body type parameter, theta, corresponding to the t-th frame video frame image _b，t Representing the complete human body posture parameter, K corresponding to the t-th frame video frame image _t Representing camera parameters corresponding to the t-th frame video frame image, J' _est，t Representing smoothed 2D bone keypoint data, θ _d，t Human body posture parameters corresponding to 2D skeleton key point data representing missing limb video frame image, theta' _m，t Posture priori constraint parameters, theta, of missing skeleton key points corresponding to t-th frame video frame image _m，t Representing human body posture parameters corresponding to key points of missing bones, N _m，t Represents the total bone key point number of the missing limb at the t-th frame, I _e，t The set of index values representing the edge-connected skeletal points of all missing limb bones of frame t, C represents the behavior class of the video sequence.

7. The method for enhancing 3D human skeleton key point data according to claim 1, wherein the step of iterating the SMPLify-X model by taking 2D skeleton key point data of the video frame image of the missing limb as input and using the posture prior constraint of the missing limb and the data enhancing algorithm to obtain the optimal 3D skeleton SMPL-X model specifically comprises:

3D human body posture estimation is carried out on the 2D bone key point data of the missing limb video frame image based on the SMPLify-X model and the gesture priori constraint and data enhancement algorithm of the missing limb, so that an SMPL-X model of the 2D bone key point data is obtained;

acquiring 3D bone key point data based on an SMPL-X model of the 2D bone key point data, and mapping the 3D bone key point data into the 2D bone key point data on an image plane through camera parameters;

matching the 2D bone key point data on the image plane with the 2D bone key point data of the missing limb video frame image to obtain a matching result;

and carrying out optimization iteration on the gesture priori constraint of the missing limb and the weight of the data enhancement algorithm based on the matching result to obtain an optimal 3D skeleton SMPL-X model.

8. A 3D human skeletal key point data enhancement system, comprising:

the data acquisition module is used for acquiring 2D skeleton key point data of the person in the missing limb video frame image;

the data comparison analysis module is used for determining 2D bone key point indexes of the missing limb edge connection and the number of the missing bone key points based on the 2D bone key point data of the missing limb video frame image and the complete 2D bone key point data;

the data smoothing module is used for carrying out frame screening and smoothing processing on the 2D skeleton key point data of the missing limb video frame image to obtain smoothed 2D skeleton key point data;

the algorithm construction module is used for constructing a gesture priori constraint and data enhancement algorithm of the missing limb based on the number of the missing bone key points, the 2D bone key point index connected with the edge of the missing limb and the video behavior category

The optimal model generation module takes the smoothed 2D skeleton key point data as input, and iterates the SMPLify-X model through the gesture priori constraint of the missing limb and the data enhancement algorithm to obtain an optimal 3D skeleton SMPL-X model;

the annotation data acquisition module is used for acquiring the 3D skeleton key point data of the optimal 3D skeleton SMPL-X model and generating a corresponding data annotation file.