Nothing Special   »   [go: up one dir, main page]

WO2022001106A1 - 关键点检测方法、装置、电子设备及存储介质 - Google Patents

关键点检测方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2022001106A1
WO2022001106A1 PCT/CN2021/075128 CN2021075128W WO2022001106A1 WO 2022001106 A1 WO2022001106 A1 WO 2022001106A1 CN 2021075128 W CN2021075128 W CN 2021075128W WO 2022001106 A1 WO2022001106 A1 WO 2022001106A1
Authority
WO
WIPO (PCT)
Prior art keywords
key point
information
key
graph model
target object
Prior art date
Application number
PCT/CN2021/075128
Other languages
English (en)
French (fr)
Inventor
金晟
刘文韬
钱晨
Original Assignee
北京市商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京市商汤科技开发有限公司 filed Critical 北京市商汤科技开发有限公司
Priority to JP2021539648A priority Critical patent/JP2022542199A/ja
Priority to KR1020217021260A priority patent/KR20220004009A/ko
Publication of WO2022001106A1 publication Critical patent/WO2022001106A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition

Definitions

  • the present disclosure relates to the technical field of image detection, and in particular, to a key point detection method, device, electronic device and storage medium.
  • Human key point detection provides high-level information support for analyzing human behavior in video, and is the basis for realizing video human action recognition and human-computer interaction.
  • human key point detection methods based on deep neural networks have become a research hotspot because of their simple and easy-to-obtain input images and efficient and accurate detection effects.
  • the detection of keypoints and the clustering of keypoints are two independent steps, that is, the clustering of keypoints is generally a non-differentiable post-processing operation.
  • the accuracy of the key point clustering process is low, that is, clustering errors may occur, and the key points of different people are clustered together as the key points of the same person, resulting in errors in the detection results.
  • embodiments of the present disclosure provide at least one key point detection method, apparatus, electronic device, and storage medium.
  • an embodiment of the present disclosure provides a key point detection method, including:
  • an image feature map and a plurality of key point heat maps are generated; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image; each of the key point heat maps The figure contains the key points of one category of the image to be detected, and the key points of different categories correspond to different parts of the target object;
  • a key point graph model is generated;
  • the key point graph model includes information of different types of key points in the to-be-detected image and information of connecting edges, each A connecting edge is an edge between two keypoints of different categories;
  • each key point belonging to the same target object is determined.
  • a key point graph model corresponding to the image to be detected can be generated based on the generated image feature map and multiple key point heat maps. Since the key point graph model includes the information in the image feature map and the key point heat map, and The image feature map can represent the relative positional relationship between different target objects in the image to be detected, so that the key points of different target objects can be more accurately distinguished based on the key point graph model, so as to improve the accuracy of key point clustering. Spend.
  • each key point belonging to the same target object is determined, including:
  • each key point belonging to the same target object is determined.
  • determining each key point belonging to the same target object based on the determined correlation degree includes: using each key point whose corresponding correlation degree is greater than a set threshold as the key point of the same target object. .
  • based on the information of each key point in the key point graph model and the information of the connecting edge determine the correlation between two key points that have a connection relationship in the key point graph model. , including: for each key point, based on the information of the key point, and the information of other key points in the key point graph model that have a connection relationship with the key point, determine the fusion feature of the key point; based on The fusion features corresponding to each key point respectively determine the degree of correlation between two key points with a connection relationship in the key point graph model.
  • the information of the key points includes location information, category information, and pixel feature information; the information of each key point in the key point graph model is determined according to the following steps: based on the key point heat map , determine the position information of each key point; based on the position information of each key point, extract the pixel feature information of the key point from the image feature map, and based on the category of the key point heat map to which the key point belongs label, to determine the category information corresponding to the key point.
  • each connecting edge in the key point graph model is generated according to the following steps: based on the category information corresponding to each key point, each key point is different from the category to which the key point belongs. The other keypoints are connected to form the connected edges in the keypoint graph model.
  • each connection edge in the key point graph model is generated according to the following steps: based on the category information corresponding to each key point and the preset matching relationship between different categories, each The key point is connected with the key point corresponding to the target category that matches the category to which the key point belongs to form a connection edge in the key point graph model.
  • the method further includes: determining the key points of the target object based on the information of each key point corresponding to each target object. type of behavior.
  • the method further includes: generating information for the target object based on the information of each key point corresponding to each target object. special effect information.
  • an embodiment of the present disclosure provides a key point detection device, including:
  • an acquisition module configured to acquire the image to be detected
  • a first generating module configured to generate an image feature map and a plurality of key point heatmaps based on the to-be-detected image; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image;
  • Each of the key point heatmaps includes a type of key points of the image to be detected, and key points of different types correspond to different parts of the target object;
  • the second generation module is configured to generate a keypoint graph model based on the image feature map and a plurality of the keypoint heatmaps;
  • the keypoint graph model includes information of different categories of keypoints in the to-be-detected image And the information of connecting edges, each connecting edge is an edge between two key points of different categories;
  • the determining module is configured to determine each key point belonging to the same target object based on the key point graph model.
  • an embodiment of the present disclosure provides an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the The processor and the memory communicate through a bus, and when the machine-readable instructions are executed by the processor, the steps of the keypoint detection method according to the first aspect or any one of the implementation manners are performed.
  • an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor as described in the first aspect or any one of the implementation manners above. The steps of the key point detection method.
  • Embodiments of the present disclosure provide a computer program product, including computer-readable code, when the computer-readable code is executed in an electronic device, the processor in the electronic device executes the code for implementing one or more of the foregoing implementations
  • the server executes the above method.
  • FIG. 1 is a schematic flowchart of a key point detection method provided by an embodiment of the present disclosure
  • FIG. 2A is an example diagram of a key point graph model provided by an embodiment of the present disclosure
  • 2B is a system block diagram of a key point detection method provided by an embodiment of the present disclosure.
  • FIG. 2C is an application scenario diagram of a key point detection method provided by an embodiment of the present disclosure.
  • FIG. 3 is a schematic structural diagram of a key point detection apparatus provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic structural diagram of an electronic device 400 according to an embodiment of the present disclosure.
  • a deep neural network can be used to detect the image to determine the key point information of the target object included in the image, wherein the key point detection method based on the deep neural network includes the detection of key points and the clustering of key points.
  • the detection of keypoints and the clustering of keypoints are two independent steps, that is, the clustering of keypoints is generally a non-differentiable post-processing operation.
  • the accuracy of the key point clustering process is low, that is, a clustering error may occur, and the key points of different people are clustered together as the key points of the same person, thereby lead to errors in the detection results.
  • an embodiment of the present disclosure provides a key point detection method.
  • the execution body of the key point detection method provided by the embodiment of the present disclosure may be a server, and the server may be a local server or a cloud server; or, the execution body of the method may also be a terminal device, for example, the terminal device may be Mobile phones, tablets, AR glasses, etc.
  • the method includes S101-S104, wherein:
  • S102 based on the image to be detected, generate an image feature map and a plurality of key point heat maps; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image; each key-point heat map contains the to-be-detected image A category of key points, and different categories of key points correspond to different parts of the target object.
  • the key point graph model includes information of key points of different categories in the image to be detected and information of connecting edges, and each connecting edge is two Edges between keypoints of different classes.
  • each key point belonging to the same target object is determined based on the key point graph model.
  • a keypoint map model corresponding to the image to be detected can be generated based on the generated image feature map and multiple keypoint heatmaps. Since the keypoint map model includes the information in the image feature map and the keypoint heatmap, and The image feature map can represent the relative positional relationship between different target objects in the image to be detected, so that the key points of different target objects can be more accurately distinguished based on the key point graph model, so as to improve the accuracy of key point clustering. Spend.
  • the image to be detected can be any image including the target object.
  • the to-be-detected image may be acquired from the connected storage device, or the to-be-detected image acquired in real time may be acquired from the connected camera device.
  • the acquired image to be detected can be input into the trained keypoint detection neural network to generate an image feature map and multiple keypoint heatmaps; and based on the image feature map, multiple keypoint heatmaps, and The trained keypoint detection neural network determines each keypoint of each target object.
  • each keypoint heatmap contains keypoints of one category of the image to be detected, and keypoints of different categories correspond to different parts of the target object.
  • the categories of key points can be head, neck, hand, etc., and then the key point heat map can be an image containing head key points, or the key point heat map can be an image containing neck key points, etc.;
  • the categories of the key points may be the set first category, the second category, etc., wherein the first category of key points may be the key points on the thumb, the second category of key points may be the key points on the index finger, etc., and then the key points
  • the point heatmap may be an image containing keypoints of the first category, or the keypoint heatmap may be an image containing keypoints of the second category, or the like.
  • the categories of key points and the number of categories can be set according to actual needs.
  • the number of key points corresponding to each target object may be set according to actual needs, for example, the number of key points corresponding to each target object may be 17, 105, and so on.
  • the number of keypoint heatmaps is consistent with the set number of keypoint categories. For example, if the set number of keypoint categories is 17, the number of keypoint heatmaps generated based on the image to be detected is also 17. Among them, the number of key points of each category can be one or more.
  • the number of image feature maps can be one or more.
  • the image feature map can represent the relative positional relationship between parts of each target object in the image to be detected and corresponding to the key points of various categories.
  • the number of image feature maps and the number of key point heat maps can be the same, that is, each image feature map can represent a type of key points of each target object in the image to be detected The relative positional relationship between the corresponding parts.
  • the size of the image feature map is consistent with the size of the keypoint heatmap.
  • image feature maps and multiple keypoint heatmaps can be obtained by setting different loss functions in the keypoint detection neural network.
  • each key point can be extracted from multiple key point heat maps and image feature maps, and each key point containing information is used as a node, and the edges between different types of key points are used as connecting edges to form Keypoint graph model.
  • the key point information may include location information, category information, and pixel feature information.
  • the information of each key point in the key point graph model can be determined according to the following steps: based on the key point heat map, determine the position information of each key point; Pixel feature information, and based on the category label of the keypoint heatmap to which the keypoint belongs, determine the category information corresponding to the keypoint.
  • the position information of each key point may be determined based on the pixel value of each pixel point in the key point heat map.
  • a pixel point with a maximum pixel value may be selected as a key point, and the position information of the selected pixel point may be determined as the position information of the key point.
  • the pixel value of a pixel in the key point heat map is greater than the pixel value of the surrounding pixels, the pixel value of the pixel is considered to be a maximum value, and the pixel is a key point.
  • the pixel value of the pixel point corresponding to the position information can be extracted from the image feature map, and the extracted pixel value is determined as the pixel characteristic information of the key point.
  • the category information corresponding to the key point can also be determined according to the category label of the key point heat map to which each key point belongs. For example, if the category label of keypoint heatmap A is head, the category information of each keypoint included in keypoint heatmap A is the head keypoint; the category label of keypoint heatmap B is neck In the case of , the category information of each key point included in the key point heatmap B is the neck key point.
  • each connecting edge in the keypoint graph model can be generated in the following two ways:
  • Mode 1 Based on the category information corresponding to each key point, each key point is connected with other key points of a different category from the key point to form a connection edge in the key point graph model.
  • Method 2 Based on the category information corresponding to each key point and the preset matching relationship between different categories, connect each key point with the key point corresponding to the target category matching the category to which the key point belongs to form a key point graph model connecting edges in .
  • the key points with the same category information are not connected, and the key points with different category information are connected to form the connection edge in the key point graph model.
  • each key point is connected with other key points of a different category from the key point to form a connection edge in the key point graph model, and then the key point graph model is obtained, which is The subsequent determination of each key point of each target object provides data support.
  • the matching relationship between different categories can be preset in advance based on the human body structure.
  • the preset matching relationship between different categories can be that the head category matches the neck category, and the neck category points are The left shoulder category, the right shoulder category, and the head category are matched, the foot category is matched with the knee category, and so on.
  • the matching relationship between different categories can be set according to actual needs.
  • a target category matching the category of the key point can be determined, and the key point is connected with the key point corresponding to the target category to form a connection edge in the key point graph model.
  • the matching relationship between different categories can be preset, for example, the head key point can be preset to match the neck key point, the foot key point can be matched with the knee key point, etc. It is connected with the key points corresponding to the target category that matches the category to which the key points belong, so that each key point is not connected with the unmatched key points, which can reduce the amount of calculation for calculating the correlation, thereby improving the efficiency of key point detection.
  • the figure includes a key point graph model 21 generated based on the first method and a key point graph model 22 generated based on the second method.
  • the figure includes key points 201 of the first category, key points 202 of the second category, and key points 203 of the third category, and also includes connecting edges 204 between different key points.
  • the key point graph model 21 includes the connection edges between various key points of different categories;
  • the key point graph model 22 includes the connection edges between different key points that have a matching relationship, and it can be seen from the figure that the preset
  • the matching relationship between the different categories is: the first category matches the second category, and the second category matches the third category.
  • each key point included in the key point graph model can be divided based on the key point graph model, and multiple key points belonging to the same target object can be divided together, and then each target object included in the image to be detected can be obtained. Corresponding multiple key points. Among them, the number of key points corresponding to each target object is the same.
  • each key point belonging to the same target object is determined, which may include:
  • A1 based on the information of each key point in the key point graph model and the information of the connecting edges, determine the degree of correlation between two key points that have a connection relationship in the key point graph model.
  • A2 based on the determined relevancy, determine each key point belonging to the same target object.
  • the correlation between two key points with a connection relationship in the key point graph model can be determined based on the information of each key point and the information of the connecting edge, because the correlation can represent the corresponding two key points.
  • the probability of belonging to the same target object between them, so the determined correlation degree can be used to realize the clustering of key points belonging to the same target object, and obtain each key point corresponding to each target object.
  • step A1 for each connecting edge in the key point graph model, the correlation between the two key points corresponding to the connecting edge can be determined, that is, the correlation corresponding to each connecting edge can be obtained.
  • the fusion feature of the key point is determined based on the information of the key point and the information of other key points that have a connection relationship with the key point in the key point graph model.
  • the fusion feature of the key point can be determined based on the information of the key point and the information of other key points in the key point graph model that have a connection relationship with the key point, wherein,
  • the other key points may be key points in the key point graph model that have connecting edges with the key point.
  • the corresponding fusion feature can be determined for each key point in the key point graph model, and the correlation between two key points corresponding to each connecting edge in the key point graph model can be determined based on the fusion features corresponding to each key point.
  • the fusion feature can be the feature vector corresponding to each key point
  • the calculated similarity can be calculated by calculating the similarity between the fusion features (feature vectors) of the two key points corresponding to each connecting edge. The degree determines the degree of correlation between these two keypoints.
  • the fusion feature of the key point is determined based on the information of the key point and the information of other key points that have a connection relationship with the key point.
  • the fusion feature of the key point can not only The features that characterize the key point can also characterize the relationship between the key point and other key points, and then based on the fusion features corresponding to each key point, the correlation between two key points that have a connection relationship can be determined. , and then based on the correlation, each key point corresponding to each target object can be more accurately determined.
  • each key point can be divided based on the determined correlation degrees, and each key point belonging to the same target object can be determined. key point.
  • determining each key point belonging to the same target object based on the determined correlation degree may include: using each key point whose corresponding correlation degree is greater than a set threshold as a key point of the same target object.
  • each key point with a corresponding correlation greater than the set threshold may be regarded as the key point of the same target object based on the set correlation threshold and the preset target number of key points of each category.
  • the key points of each target object include a plurality of key points of different categories, and the number of key points of each category is consistent with the preset target number.
  • the method may further include: determining the behavior type of the target object based on the information of each key point corresponding to each target object.
  • the information of each key point of each target object can be input into the behavior detection neural network to determine the behavior type of the target object, for example, the behavior type can be For running, walking, raising arms, etc.
  • each key point belonging to the same target object based on the key point graph model may also include: based on the information of each key point corresponding to each target object, generating special effects information for the target object. .
  • the position of the target part of the target object can be determined, and based on the preset special effect information corresponding to the target part, corresponding special effect information is generated at the position of the target part of the target object.
  • the target part may be an arm, a head, a hand, or the like.
  • the arm position of the target object can be determined, and based on the preset special effect information of the arm, corresponding special effect information can be generated at the arm position of the target object.
  • the action type of the target object may also be determined according to the information of each key point corresponding to each target object, and corresponding special effect information is generated for the target object based on the preset mapping relationship between the action type and the special effect information. For example, if the action type of the target object is determined to be a heart-to-heart action based on the information of each key point corresponding to the target object A, heart-shaped special effect information can be generated for the target object.
  • the key point detection of the target object is realized by a bottom-up method for detecting key points of multiple target objects.
  • This type of method first detects all possible key points of the target object in the image, and extracts the information of each key point, such as the embedding feature of the key point; then, by solving the optimization equation, these key points are clustered together. class and assign to multiple different target objects. It can be seen that the bottom-up method runs faster and is more robust to immediate objects or occlusions.
  • Method 1 is implemented based on the keypoint embedding feature (Keypoint Embedding, KE);
  • the second method is implemented by a key point detection method based on Spatial Instance Embedding (SIE).
  • SIE Spatial Instance Embedding
  • KE mainly contains the apparent information of the pixels near the key points
  • the embedding value of each key point of the same target object can be shortened, and the key points of different target objects can be pulled away.
  • Embedding value which realizes the embedded feature extraction of key points.
  • the embedded features of key points can be extracted by returning each pixel value to the vector of the center of the target object.
  • KE mainly involves pixel feature information.
  • the pixel feature information can include the apparent information of the pixel, for example, the pixel value of the pixel, which is not sensitive to the spatial position information, and can model the long distance between key points.
  • relying only on KE may have the problem of clustering keypoints of different target objects at a distance.
  • Clustering of keypoints is a non-differentiable post-processing operation.
  • the clustering process of key points is fixed post-processing, and cannot be optimized by learning in the data; 2) Since there is no joint optimization, the clustering accuracy of key points is not high, which may cause clustering.
  • Class errors for example, clustering keypoints of different target objects as keypoints of the same target object; 3)
  • the post-clustering process requires additional hyperparameter settings.
  • FIG. 2B The system block diagram of the method is shown in FIG. 2B , wherein:
  • the keypoint clustering step is a post-processing operation after keypoint detection.
  • An embodiment of the present disclosure provides a keypoint detection method, which includes two submodules: a keypoint candidate extraction module G1 (Keypoint Candidate Proposal) and a graph clustering module G2 (Graph Grouping).
  • a keypoint candidate extraction module G1 Keypoint Candidate Proposal
  • G2 graph clustering module
  • the keypoint candidate extraction module G1 is used for: for each frame of image, firstly, through the multi-task bottom-up target object keypoint model, directly output the keypoint heatmaps (Heatmaps) of the target object keypoints, and the image feature map or Spatial instance image feature maps (Feature Maps).
  • Heatmaps keypoint heatmaps
  • Feature Maps image feature map or Spatial instance image feature maps
  • the graph clustering module G2 is used for: judging whether two key points belong to the same target object, and clustering the key points of the same target object together.
  • Figure 2C shows the process of inputting the graph convolutional network from the keypoint heatmap P1 and the image feature map P2.
  • the key point graph model G is divided into two parts: the key point V and the edge E.
  • the key point V is the information of each key point, that is, it includes "the key point category T, the key point coordinate X, and the key point feature information F" .
  • the edge V represents the relationship between key points, that is, whether they belong to the same target object.
  • EdgeConv Use edge convolution EdgeConv to build a graph convolutional neural network model, convolve the key point graph model, and continuously update the feature information of key points. Then train an edge classifier to discriminate each pair of key points to determine whether the pair of key points belong to the same target object.
  • the keypoint candidate extraction module G1 the graph clustering module G2, and the edge classifier, the training optimization can be performed end-to-end.
  • the key point detection method provided by the embodiment of the present disclosure can predict the position information of the key points of the target object on the one hand; on the other hand, can determine the behavior type of the target object; On the other hand, real-time special effect information can be added to different parts of the target object.
  • the multi-target object pose estimation can be transformed into a graph clustering problem; the topological structure information of the target object can be retained, and the feature information of the multi-target object pose graph can be extracted; a deep neural network model can be used to obtain the multi-target object
  • the clustering result of the object pose it can avoid post-processing operations such as clustering of key points in related technologies, which simplifies the operation steps; it can integrate the detection of key points and the clustering of key points, and improve the clustering of key points. precision.
  • an embodiment of the present disclosure also provides a key point detection apparatus.
  • a schematic diagram of the architecture of a key point detection apparatus provided by an embodiment of the present disclosure includes an acquisition module 301 , a first generation Module 302, second generation module 303, determination module 304, behavior type determination module 305, and special effect information generation module 306, wherein:
  • an acquisition module 301 configured to acquire an image to be detected
  • the first generation module 302 is configured to generate an image feature map and a plurality of key point heat maps based on the image to be detected; the image feature map is used to represent the relative positional relationship between each target object in the image to be detected ; Each described key point heat map includes a key point of a category of the image to be detected, and the key points of different categories correspond to different parts of the target object;
  • the second generation module 303 is configured to generate a key point graph model based on the image feature map and a plurality of the key point heat maps; the key point graph model includes the key points of different categories in the image to be detected. Information and information of connecting edges, each connecting edge is an edge between two key points of different categories;
  • the determining module 304 is configured to determine each key point belonging to the same target object based on the key point graph model.
  • the determining module 304 in the case of determining each key point belonging to the same target object based on the key point graph model, is configured as:
  • each key point belonging to the same target object is determined.
  • the determining module 304 in the case of determining each key point belonging to the same target object based on the determined correlation degree, is configured to:
  • Each key point whose correlation degree is greater than the set threshold is regarded as the key point of the same target object.
  • the determining module 304 determines, based on the information of each key point in the key point graph model and the information of the connecting edge, two of the key point graph models that have a connection relationship.
  • the configuration is:
  • the fusion feature of the key point is determined based on the information of the key point and the information of other key points in the key point graph model that have a connection relationship with the key point;
  • the correlation between the two key points with a connection relationship in the key point graph model is determined.
  • the information of the key points includes location information, category information, and pixel feature information
  • the second generation module 303 is configured to determine the information of each key point in the key point graph model according to the following steps:
  • the second generation module 303 is configured to generate each connection edge in the key point graph model according to the following steps:
  • each key point is connected with other key points of a different category from the key point to form a connection edge in the key point graph model.
  • the second generation module 303 is configured to generate each connection edge in the key point graph model according to the following steps:
  • each key point is connected with the key point corresponding to the target category matching the category to which the key point belongs to form the key point. Connected edges in a dot graph model.
  • the method further includes:
  • the behavior type determination module 305 is configured to determine the behavior type of the target object based on the information of each key point corresponding to each target object.
  • the method further includes:
  • the special effect information generating module 306 is configured to generate special effect information for the target object based on the information of each key point corresponding to each target object.
  • the functions or templates included in the apparatus provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments.
  • reference may be made to the descriptions in the above method embodiments. Repeat.
  • an embodiment of the present disclosure also provides an electronic device.
  • a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure includes a processor 401 , a memory 402 , and a bus 403 .
  • the memory 402 is configured to store execution instructions, including the memory 4021 and the external memory 4022; the memory 4021 here is also called the internal memory, and is configured to temporarily store the operation data in the processor 401 and the data exchanged with the external memory 4022 such as the hard disk,
  • the processor 401 exchanges data with the external memory 4022 through the memory 4021.
  • the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the following instructions:
  • an image feature map and a plurality of key point heat maps are generated; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image; each of the key point heat maps The figure contains the key points of one category of the image to be detected, and the key points of different categories correspond to different parts of the target object;
  • a key point graph model is generated;
  • the key point graph model includes information of different types of key points in the to-be-detected image and information of connecting edges, each A connecting edge is an edge between two keypoints of different categories;
  • each key point belonging to the same target object is determined.
  • an embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the key point detection described in the foregoing method embodiments is executed steps of the method.
  • the computer program product of the key point detection method provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program codes, and the instructions included in the program code can be used to execute the key point detection methods described in the above method embodiments. For the steps, reference may be made to the above method embodiments, which will not be repeated here.
  • the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that contribute to the prior art or the parts of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .
  • the present disclosure generates a keypoint map model corresponding to the image to be detected based on the generated image feature map and multiple keypoint heatmaps. Since the keypoint map model includes information in the image feature map and the keypoint heatmap, the image feature map The relative positional relationship between different target objects in the image to be detected can be characterized, so that the key points of different target objects can be more accurately distinguished based on the key point graph model, so as to improve the accuracy of key point clustering.

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Storage Device Security (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)

Abstract

本公开提供了一种关键点检测方法、装置、电子设备及存储介质，所述方法包括: 获取待检测图像; 基于所述待检测图像，生成图像特征图和多个关键点热图; 所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系; 每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位; 基于所述图像特征图和多个所述关键点热图，生成关键点图模型; 所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边; 基于所述关键点图模型，确定属于同一目标对象的各个关键点。

Description

关键点检测方法、装置、电子设备及存储介质
相关申请的交叉引用
本公开基于申请号为202010622132.3、申请日为2020年06月30日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本公开。
技术领域
本公开涉及图像检测技术领域，尤其涉及一种关键点检测方法、装置、电子设备及存储介质。
背景技术
人体关键点检测为分析视频中人的行为提供了高层信息支持，是实现视频人体动作识别和人机交互的基础。近年来，基于深度神经网络的人体关键点检测方法，因其输入图像简单易获取、检测效果高效准确，成为了研究热点。
一般地，基于深度神经网络的人体关键点检测方法中，关键点的检测和关键点的聚类是两个独立的步骤，即关键点的聚类一般是不可微分的后处理操作。但是，这种方式下，关键点聚类过程的准确度较低，即可能会产生聚类错误，将不同人的关键点作为同一个人的关键点聚类在一起，从而导致检测结果出现错误。
发明内容
有鉴于此，本公开实施例至少提供一种关键点检测方法、装置、电子设备及存储介质。
第一方面，本公开实施例提供了一种关键点检测方法，包括：
获取待检测图像；
基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
基于所述图像特征图和多个所述关键点热图，生成关键点图模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
基于所述关键点图模型，确定属于同一目标对象的各个关键点。
采用上述方法，可以基于生成的图像特征图和多个关键点热图，生成待检测图像对应的关键点图模型，由于关键点图模型中包括图像特征图和关键点热图中的信息，而图像特征图可以表征出待检测图像中不同目标对象之间的相对位置关系，从而使得基于关键点图模型，可以较准确地对不同目标对象的关键点进行区分，以提高关键点聚类的精准度。
一种可能的实施方式中，基于所述关键点图模型，确定属于同一目标对象的各个关键点，包括：
基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度；
基于确定的所述相关度，确定属于同一目标对象的各个关键点。
一种可能的实施方式中，基于确定的所述相关度，确定属于同一目标对象的各个关键点，包括：将对应的所述相关度大于设定阈值的各个关键点作为同一目标对象的关键点。
一种可能的实施方式中，基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度，包括：针对每个关键点，基于所述关键点的信息，和所述关键点图模型中与所述关键点存在连接关系的其它关键点的信息，确定所述关键点的融合特征；基于各个关键点分别对应的融合特征，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度。
一种可能的实施方式中，所述关键点的信息包括位置信息、类别信息、以及像素特征信息；根据以下步骤确定所述关键点图模型中各个关键点的信息：基于所述关键点热图，确定各个关键点的位置信息；基于每个所述关键点的位置信息，从所述图像特征图中提取所述关键点的像素特征信息，并基于所述关键点所属关键点热图的类别标签，确定所述关键点对应的类别信息。
一种可能的实施方式中，根据以下步骤生成所述关键点图模型中的各个连接边：基于各个关键点对应的所述类别信息，将每个关键点和与所述关键点所属的类别不同的其它关键点连接，形成所述关键点图模型中的连接边。
一种可能的实施方式中，根据以下步骤生成所述关键点图模型中的各个连接边：基于各个关键点对应的所述类别信息、和预设的不同类别之间的匹配关系，将每个关键点和与所述关键点所属类别匹配的目标类别对应的关键点连接，形成所述关键点图模型中的连接边。
一种可能的实施方式中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：基于每个目标对象对应的各个关键点的信息，确定所述目标对象的行为类型。
一种可能的实施方式中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：基于每个目标对象对应的各个关键点的信息，生成针对所述目标对象的特效信息。
以下装置、电子设备等的效果描述参见上述方法的说明，这里不再赘述。
第二方面，本公开实施例提供了一种关键点检测装置，包括：
获取模块，配置为获取待检测图像；
第一生成模块，配置为基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
第二生成模块，配置为基于所述图像特征图和多个所述关键点热图，生成关键点图 模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
确定模块，配置为基于所述关键点图模型，确定属于同一目标对象的各个关键点。
第三方面，本公开实施例提供一种电子设备，包括：处理器、存储器和总线，所述存储器存储有所述处理器可执行的机器可读指令，在电子设备运行的情况下，所述处理器与所述存储器之间通过总线通信，所述机器可读指令被所述处理器执行时执行如上述第一方面或任一实施方式所述的关键点检测方法的步骤。
第四方面，本公开实施例提供一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行如上述第一方面或任一实施方式所述的关键点检测方法的步骤。
本公开实施例提供了一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现上述一个或多个实施例中服务器执行上述方法。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，此处的附图被并入说明书中并构成本说明书中的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。应当理解，以下附图仅示出了本公开的某些实施例，因此不应被看作是对范围的限定，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他相关的附图。
图1为本公开实施例所提供的一种关键点检测方法的流程示意图；
图2A为本公开实施例所提供的一种关键点图模型的示例图；
图2B为本公开实施例所提供的一种关键点检测方法的系统框图；
图2C为本公开实施例所提供的一种关键点检测方法的应用场景图；
图3为本公开实施例所提供的一种关键点检测装置的架构示意图；
图4为本公开实施例所提供的一种电子设备400的结构示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。通常在此处附图中描述和示出的本公开实施例的组件可以以各种不同的配置来布置和设计。因此，以下对在附图中提供的本公开的实施例的详细描述并非旨在限制要求保护的本公开的范围，而是仅仅表示本公开的选定实施例。基于本公开的实施例，本领域技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都属于本公开保护的范围。
一般地，可以使用深度神经网络对图像进行检测，确定图像中包括的目标对象的关 键点信息，其中，基于深度神经网络的关键点检测方法中，包括关键点的检测和关键点的聚类。
目前，关键点的检测和关键点的聚类是两个独立的步骤，即关键点的聚类一般是不可微分的后处理操作。但是，在使用这种方式的情况下，使得关键点聚类过程的准确度较低，即可能会产生聚类错误，将不同人的关键点作为同一个人的关键点而聚类在一起，从而导致检测结果出现错误。
为了解决上述问题，本公开实施例提供了一种关键点检测方法。
为便于对本公开实施例进行理解，首先对本公开实施例所公开的一种关键点检测方法进行详细介绍。
本公开实施例提供的关键点检测方法的执行主体可以为服务器，该服务器可以为本地服务器，也可以为云端服务器；或者，该方法的执行主体也可以为终端设备，比如，该终端设备可以为手机、平板电脑、AR眼镜等。
参见图1所示，为本公开实施例所提供的一种关键点检测方法的流程示意图，该方法包括S101-S104，其中：
S101，获取待检测图像。
S102，基于待检测图像，生成图像特征图和多个关键点热图；图像特征图用于表征待检测图像中各个目标对象之间的相对位置关系；每个关键点热图中包含待检测图像的一种类别的关键点，不同类别的关键点对应目标对象的不同部位。
S103，基于图像特征图和多个关键点热图，生成关键点图模型；关键点图模型中包含待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边。
S104，基于关键点图模型，确定属于同一目标对象的各个关键点。
上述方法中，可以基于生成的图像特征图和多个关键点热图，生成待检测图像对应的关键点图模型，由于关键点图模型中包括图像特征图和关键点热图中的信息，而图像特征图可以表征出待检测图像中不同目标对象之间的相对位置关系，从而使得基于关键点图模型，可以较准确地对不同目标对象的关键点进行区分，以提高关键点聚类的精准度。
针对S101以及S102：
待检测图像可以为任一包括目标对象的图像。在实施过程中，可以从相连的存储装置中获取待检测图像，也可以为从相连的摄像装置中获取实时采集到的待检测图像。
在实施过程中，可以将获取的待检测图像输入至训练后的关键点检测神经网络中，生成图像特征图和多个关键点热图；并基于图像特征图、多个关键点热图、以及训练后的关键点检测神经网络，确定每个目标对象的各个关键点。
这里，每个关键点热图中包含待检测图像的一种类别的关键点，不同类别的关键点对应目标对象的不同部位。比如，关键点的类别可以为头部、颈部、手部等，进而关键点热图可以为包含头部关键点的图像，或者，关键点热图可以为包含颈部关键点的图像等；或者，关键点的类别可以为设置的第一类别、第二类别等，其中，第一类别关键点可以为拇指上的关键点，第二类别关键点可以为食指上的关键点等，进而关键点热图可以为包含第一类别关键点的图像，或者，关键点热图可以为包含第二类别关键点的图像 等。其中，关键点的类别和类别的数量可以根据实际需要进行设置。以及，每个目标对象对应的关键点的数量可以根据实际需要进行设置，比如，每个目标对象对应的关键点的数量可以为17个、105个等。
这里，关键点热图的数量与设置的关键点类别的数量一致，比如，若设置的关键点的类别数量为17个，则基于待检测图像生成的关键点热图的数量也为17个。其中，每种类别的关键点的数量可以为一个或多个。
图像特征图的数量可以为一个，也可以为多个。其中，在图像特征图的数量为一个的情况下，则该图像特征图可以表征待检测图像中各个目标对象的、各种类别的关键点对应的部位之间的相对位置关系。在图像特征图的数量为多个的情况下，图像特征图的数量与关键点热图的数量可以相同，即每张图像特征图可以表征待检测图像中各个目标对象的一种类别的关键点对应的部位之间的相对位置关系。其中，图像特征图的尺寸与关键点热图的尺寸一致。
在实施过程中，可以通过在关键点检测神经网络中设置不同的损失函数，得到图像特征图和多个关键点热图。
针对S103：
这里，可以从多个关键点热图和图像特征图中提取得到每个关键点的信息，将包含信息的每个关键点作为节点、以不同类别的关键点之间的边作为连接边，构成了关键点图模型。
一种可选实施方式中，关键点的信息可以包括位置信息、类别信息、以及像素特征信息。其中，可以根据以下步骤确定关键点图模型中各个关键点的信息：基于关键点热图，确定各个关键点的位置信息；基于每个关键点的位置信息，从图像特征图中提取关键点的像素特征信息，并基于关键点所属关键点热图的类别标签，确定关键点对应的类别信息。
在实施过程中，可以基于关键点热图中每个像素点的像素值，确定各个关键点的位置信息。示例性的，针对每个关键点热图，可以选择像素值为极大值的像素点，确定为一关键点，并将选择的该像素点的位置信息确定为关键点的位置信息。其中，若关键点热图中某一像素点的像素值大于周围像素点的像素值，则认为该像素点的像素值为极大值，该像素点为关键点。
在得到了每个像素点的位置信息之后，可以从图像特征图中提取与该位置信息对应的像素点的像素值，将提取的像素值确定为关键点的像素特征信息。
同时，还可以根据每个关键点所属关键点热图的类别标签，确定关键点对应的类别信息。比如，在关键点热图A的类别标签为头部的情况下，关键点热图A中包括的各个关键点的类别信息为头部关键点；在关键点热图B的类别标签为颈部的情况下，关键点热图B中包括的各个关键点的类别信息为颈部关键点。
一种可选实施方式中，可以根据以下两种方式生成关键点图模型中的各个连接边：
方式一、基于各个关键点对应的类别信息，将每个关键点和与关键点所属的类别不同的其它关键点连接，形成关键点图模型中的连接边。
方式二、基于各个关键点对应的类别信息、和预设的不同类别之间的匹配关系，将每个关键点和与关键点所属类别匹配的目标类别对应的关键点连接，形成关键点图模型 中的连接边。
方式一中，可以基于每个关键点对应的类别信息，对类别信息相同的关键点不进行连接，对类别信息不同的关键点相连接，形成关键点图模型中的连接边。
这里，通过基于各个关键点对应的类别信息，将每个关键点和与关键点所属的类别不同的其它关键点连接，形成关键点图模型中的连接边，进而得到了关键点图模型，为后续确定每个目标对象的各个关键点提供了数据支持。
方式二中，可以基于人体结构，提前预设不同类别之间的匹配关系，比如，预设的不同类别之间的匹配关系可以为头部类别与颈部类别相匹配，颈部类别点分别与左肩类别、右肩类别、以及头部类别相匹配，脚部类别与膝部类别相匹配等。其中，不同类别之间的匹配关系可以根据实际需要进行设置。
进而针对每个关键点，可以确定与该关键点的类别匹配的目标类别，将该关键点与目标类别对应的关键点连接，形成了关键点图模型中的连接边。
在上述实施方式中，可以预设不同类别之间的匹配关系，比如，可以预设头部关键点与颈部关键点匹配，脚部关键点与膝部关键点匹配等，将每个关键点和与关键点所属类别匹配的目标类别对应的关键点连接，使得每个关键点与不匹配的关键点不进行连接，可以减少计算相关度的计算量，进而提高关键点检测的效率。
参见图2A所示的一种关键点图模型的示例图，该图中包括基于方式一生成的关键点图模型21、和基于方式二生成的关键点图模型22。其中，图中包括第一类别的关键点201、第二类别的关键点202、以及第三类别的关键点203，还包括不同关键点之间的连接边204。由图可知，关键点图模型21中，包括不同类别的各个关键点之间的连接边；关键点图模型22中，包括存在匹配关系的不同关键点之间的连接边，由图可知预设的不同类别之间的匹配关系为：第一类别与第二类别匹配，第二类别与第三类别匹配。
针对S104：
这里，可以基于关键点图模型，对关键点图模型中包括的各个关键点进行划分，将属于同一目标对象的多个关键点划分在一起，进而可以得到待检测图像中包括的每个目标对象对应的多个关键点。其中，每个目标对象对应的关键点的数量相同。
一种可选实施方式中，基于关键点图模型，确定属于同一目标对象的各个关键点，可以包括：
A1，基于关键点图模型中各个关键点的信息以及连接边的信息，确定关键点图模型中存在连接关系的两个关键点之间的相关度。
A2，基于确定的相关度，确定属于同一目标对象的各个关键点。
上述实施方式下，可以基于各个关键点的信息以及连接边的信息，确定关键点图模型中存在连接关系的两个关键点之间的相关度，由于该相关度可以表征对应的两个关键点之间属于同一目标对象的概率，故可以通过确定的相关度，实现将属于同一目标对象的各个关键点聚类在一起，得到每个目标对象对应的各个关键点。
在步骤A1中，可以针对关键点图模型中的每一连接边，确定该连接边对应的两个关键点之间的相关度，即可以得到每条连接边对应的相关度。
作为一可选实施方式，基于关键点图模型中各个关键点的信息以及连接边的信息，确定关键点图模型中存在连接关系的两个关键点之间的相关度，包括：
一、针对每个关键点，基于关键点的信息，和关键点图模型中与关键点存在连接关系的其它关键点的信息，确定关键点的融合特征。
二、基于各个关键点分别对应的融合特征，确定关键点图模型中存在连接关系的两个关键点之间的相关度。
在确定每个关键点的融合特征的情况下，可以基于该关键点的信息、和关键点图模型中与该关键点存在连接关系的其他关键点的信息，确定关键点的融合特征，其中，其他关键点可以为关键点图模型中与该关键点之间存在连接边的关键点。
这里，可以为关键点图模型中的每个关键点确定对应的融合特征，基于各个关键点分别对应的融合特征，确定关键点图模型中每条连接边对应的两个关键点之间的相关度。
在实施过程中，融合特征可以为每个关键点对应的特征向量，则可以通过计算每条连接边对应的两个关键点的融合特征(特征向量)之间的相似度，将计算得到的相似度确定这两个关键点之间的相关度。
上述实施方式下，针对每个关键点，基于该关键点的信息和与该关键点存在连接关系的其它关键点的信息，确定该关键点的融合特征，这样，该关键点的融合特征不仅可以表征该关键点的特征，还可以表征该关键点与其他关键点之间的关联关系，进而基于各个关键点分别对应的融合特征，可以确定出存在连接关系的两个关键点之间的相关度，进而可以基于该相关度，较准确的确定每个目标对象对应的各个关键点。
在步骤A2中，在确定得到关键点图模型中每条连接边对应的两个关键点的相关度之后，可以基于确定的各个相关度，将各个关键点进行划分，确定属于同一目标对象的各个关键点。
作为一可选实施方式，基于确定的相关度，确定属于同一目标对象的各个关键点，可以包括：将对应的相关度大于设定阈值的各个关键点作为同一目标对象的关键点。
在实施过程中，可以基于设置相关度的阈值、和预设的每种类别的关键点的目标数量，将对应的相关度大于设置的阈值的各个关键点作为同一目标对象的关键点。其中，每个目标对象的关键点中包括多个不同类别的关键点，以及每个类别的关键点的数量预设的目标数量相符。
一种可选实施方式中，在基于关键点图模型，确定属于同一目标对象的各个关键点之后，还可以包括：基于每个目标对象对应的各个关键点的信息，确定目标对象的行为类型。
这里，在得到每个目标对象的各个关键点的信息之后，可以将每个目标对象的各个关键点的信息输入至行为检测神经网络中，确定该目标对象的行为类型，比如，该行为类型可以为跑步、走步、托举双臂等。
一种可选实施方式中，在基于关键点图模型，确定属于同一目标对象的各个关键点之后，还可以包括：基于每个目标对象对应的各个关键点的信息，生成针对目标对象的特效信息。
这里，可以针对目标对象的各个关键点的信息，确定目标对象的目标部位的位置，基于预设的目标部位对应的特效信息，在目标对象的目标部位的位置处生成对应的特效信息。其中，目标部位可以为手臂、头部、手部等。比如，可以针对目标对象的各个关 键点的信息，确定目标对象的手臂位置，并基于预设的手臂的特效信息，在目标对象的手臂位置处生成对应的特效信息。
这里，还可以针对每个目标对象对应的各个关键点的信息，确定该目标对象的动作类型，基于预设的动作类型与特效信息之间的映射关系，为该目标对象生成对应的特效信息。比如，若基于目标对象A对应的各个关键点的信息，确定该目标对象的动作类型为比心动作，则可以为该目标对象生成心形的特效信息。
相关技术中，对目标对象进行关键点检测通过自底向上的多个目标对象关键点检测方法实现。这类方法首先将图片中的所有可能的目标对象关键点检测出来，同时提取每个关键点的信息，例如关键点的嵌入特征(Embedding feature)；然后，通过求解优化方程，将这些关键点聚类并分配给多个不同的目标对象。由此可见，自底向上的方法运行速度更快，且对于紧邻的目标或遮挡情况鲁棒性更好。
提取关键点的嵌入特征有以下两种方法：
方法一，基于关键点嵌入特征(Keypoint Embedding，KE)实现；
方法二，是基于空间实例嵌入特征(Spatial Instance Embedding，SIE)的关键点检测方法实现。
其中，方法一在训练过程中，由于KE主要包含了关键点附近的像素的表观信息，因此可以通过拉近同一个目标对象的各个关键点的嵌入值，拉远不同目标对象的关键点的嵌入值，实现提取关键点的嵌入特征。
其中，方法二在训练过程中，由于SIE包含了目标对象中心位置信息，因此可以通过对每一个像素值回归到目标对象中心的向量，实现提取关键点的嵌入特征。
由此可见，KE主要涉及像素特征信息，这里，所述像素特征信息可以包括像素的表观信息，例如，像素的像素值，对空间位置信息不敏感，可以建模长距离的关键点之间的关系；然而，由于缺乏空间约束，只依赖KE可能会存在将远处不同目标对象的关键点聚在一起的问题。
由此可见，相关技术中，自底向上的关键点检测方法中，关键点的检测和关键点的聚类是两个独立的步骤。关键点的聚类是一种不可微分的后处理操作。存在以下几个问题：1)关键点的聚类过程是固定的后处理，而不能在数据中学习优化的；2)由于没有进行联合优化，关键点的聚类精度不高，可能会产生聚类错误，例如，把不同目标对象的关键点，当作是同一个目标对象的关键点而聚类在一起；3)聚类后处理过程，需要额外的超参数设置。
为解决上述问题，本公开实施例提供一种关键点检测方法，所述方法的系统框图如图2B所示，其中：
相关技术中，包括两个步骤：关键点的检测S21和关键点的聚类S22。关键点的聚类步骤是在关键点检测后的一种后处理操作。
本公开实施例提供一种关键点检测方法，包括两个子模块：关键点候选提取模块G1(Keypoint Candidate Proposal)和图聚类模块G2(Graph Grouping)。
关键点候选提取模块G1用于：对于每一帧图像，首先通过多任务的自底向上的目标对象关键点模型，直接输出目标对象关键点的关键点热图(Heatmaps)，以及图像特征图或空间实例图像特征图(Feature Maps)。通过对关键点热图取极大值(argmax)， 可以得到关键点的像素坐标位置X。同时，可以从图像特征图的对应位置信息，获取关键点的特征信息F。
图聚类模块G2用于：判断两两关键点是否属于同一个目标对象，并把同一个目标对象的关键点聚类在一起。
举例说明，图2C展示了由关键点热图P1和图像特征图P2，输入图卷积网络的过程。首先，提取关键点的信息，构造关键点图模型G＝{V,E}。关键点图模型G分为关键点V和边E两部分，其中，关键点V为各个关键点的信息，即包含「关键点的类别T，关键点的坐标X，关键点的特征信息F」。而边V代表关键点之间的关系，即是否属于同一个目标对象。构造好关键点图模型之后，进行关键点相关度的计算。使用边缘卷积EdgeConv搭建图卷积神经网络模型，对关键点图模型进行卷积，不断更新关键点的特征信息。接着训练一个边分类器，对每一对关键点进行判别，判断这一对关键点是否属于同一个目标对象。通过关键点候选提取模块G1、图聚类模块G2、边分类器可以端到端地进行训练优化。
一种可能的实施方式中，本公开实施例所提供的一种关键点检测方法一方面，可以对目标对象关键点的位置信息进行预测；另一方面，可以确定所述目标对象的行为类型；再一方面，可以在目标对象的不同部位增加实时特效信息。
采用上述方法，能够将多目标对象姿态估计转化为图聚类问题；能够保留目标对象的拓扑结构信息，进行对多目标对象姿态图的特征信息提取；能够通过一个深度神经网络模型，得到多目标对象姿态的聚类结果；能够避免相关技术中，需要对关键点的进行聚类等后处理操作，简化了操作步骤；能够整合关键点的检测和关键点的聚类，提高关键点的聚类精度。
本领域技术人员可以理解，在实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的执行顺序应当以其功能和可能的内在逻辑确定。
基于相同的构思，本公开实施例还提供了一种关键点检测装置，参见图3所示，为本公开实施例提供的一种关键点检测装置的架构示意图，包括获取模块301、第一生成模块302、第二生成模块303、确定模块304、行为类型确定模块305、以及特效信息生成模块306，其中：
获取模块301，配置为获取待检测图像；
第一生成模块302，配置为基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
第二生成模块303，配置为基于所述图像特征图和多个所述关键点热图，生成关键点图模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
确定模块304，配置为基于所述关键点图模型，确定属于同一目标对象的各个关键点。
一种可能的实施方式中，所述确定模块304，在基于所述关键点图模型，确定属于 同一目标对象的各个关键点的情况下，配置为：
基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度；
基于确定的所述相关度，确定属于同一目标对象的各个关键点。
一种可能的实施方式中，所述确定模块304，在基于确定的所述相关度，确定属于同一目标对象的各个关键点的情况下，配置为：
将对应的所述相关度大于设定阈值的各个关键点作为同一目标对象的关键点。
一种可能的实施方式中，所述确定模块304，在基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度的情况下，配置为：
针对每个关键点，基于所述关键点的信息，和所述关键点图模型中与所述关键点存在连接关系的其它关键点的信息，确定所述关键点的融合特征；
基于各个关键点分别对应的融合特征，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度。
一种可能的实施方式中，所述关键点的信息包括位置信息、类别信息、以及像素特征信息；
所述第二生成模块303，配置为根据以下步骤确定所述关键点图模型中各个关键点的信息：
基于所述关键点热图，确定各个关键点的位置信息；
基于每个所述关键点的位置信息，从所述图像特征图中提取所述关键点的像素特征信息，并基于所述关键点所属关键点热图的类别标签，确定所述关键点对应的类别信息。
一种可能的实施方式中，所述第二生成模块303，配置为根据以下步骤生成所述关键点图模型中的各个连接边：
基于各个关键点对应的所述类别信息，将每个关键点和与所述关键点所属的类别不同的其它关键点连接，形成所述关键点图模型中的连接边。
一种可能的实施方式中，所述第二生成模块303，配置为根据以下步骤生成所述关键点图模型中的各个连接边：
基于各个关键点对应的所述类别信息、和预设的不同类别之间的匹配关系，将每个关键点和与所述关键点所属类别匹配的目标类别对应的关键点连接，形成所述关键点图模型中的连接边。
一种可能的实施方式中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
行为类型确定模块305，配置为基于每个目标对象对应的各个关键点的信息，确定所述目标对象的行为类型。
一种可能的实施方式中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
特效信息生成模块306，配置为基于每个目标对象对应的各个关键点的信息，生成针对所述目标对象的特效信息。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模板可以用于执行 上文方法实施例描述的方法，其实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
基于同一技术构思，本公开实施例还提供了一种电子设备。参照图4所示，为本公开实施例提供的电子设备的结构示意图，包括处理器401、存储器402、和总线403。其中，存储器402配置为存储执行指令，包括内存4021和外部存储器4022；这里的内存4021也称内存储器，配置为暂时存放处理器401中的运算数据，以及与硬盘等外部存储器4022交换的数据，处理器401通过内存4021与外部存储器4022进行数据交换，在电子设备400运行的情况下，处理器401与存储器402之间通过总线403通信，使得处理器401在执行以下指令：
获取待检测图像；
基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
基于所述图像特征图和多个所述关键点热图，生成关键点图模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
基于所述关键点图模型，确定属于同一目标对象的各个关键点。
此外，本公开实施例还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行上述方法实施例中所述的关键点检测方法的步骤。
本公开实施例所提供的关键点检测方法的计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令可用于执行上述方法实施例中所述的关键点检测方法的步骤，可参见上述方法实施例，在此不再赘述。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统和装置的工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。在本公开所提供的几个实施例中，应所述理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，又例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些通信接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个处理器可执行的非易失的计算机可读取存储介质中。基于这样的理解，本公开 的技术方案本质上或者说对现有技术做出贡献的部分或者所述技术方案的部分可以以软件产品的形式体现出来，所述计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以权利要求的保护范围为准。
工业实用性
本公开基于生成的图像特征图和多个关键点热图，生成待检测图像对应的关键点图模型，由于关键点图模型中包括图像特征图和关键点热图中的信息，而图像特征图可以表征出待检测图像中不同目标对象之间的相对位置关系，从而使得基于关键点图模型，可以较准确地对不同目标对象的关键点进行区分，以提高关键点聚类的精准度。

Claims (21)

  1. 一种关键点检测方法，包括：
    获取待检测图像；
    基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
    基于所述图像特征图和多个所述关键点热图，生成关键点图模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
    基于所述关键点图模型，确定属于同一目标对象的各个关键点。
  2. 根据权利要求1所述的方法，其中，基于所述关键点图模型，确定属于同一目标对象的各个关键点，包括：
    基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度；
    基于确定的所述相关度，确定属于同一目标对象的各个关键点。
  3. 根据权利要求2所述的方法，其中，基于确定的所述相关度，确定属于同一目标对象的各个关键点，包括：
    将对应的所述相关度大于设定阈值的各个关键点作为同一目标对象的关键点。
  4. 根据权利要求2或3所述的方法，其中，基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度，包括：
    针对每个关键点，基于所述关键点的信息，和所述关键点图模型中与所述关键点存在连接关系的其它关键点的信息，确定所述关键点的融合特征；
    基于各个关键点分别对应的融合特征，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度。
  5. 根据权利要求1至4任一所述的方法，其中，所述关键点的信息包括位置信息、类别信息、以及像素特征信息；
    根据以下步骤确定所述关键点图模型中各个关键点的信息：
    基于所述关键点热图，确定各个关键点的位置信息；
    基于每个所述关键点的位置信息，从所述图像特征图中提取所述关键点的像素特征信息，并基于所述关键点所属关键点热图的类别标签，确定所述关键点对应的类别信息。
  6. 根据权利要求5所述的方法，其中，根据以下步骤生成所述关键点图模型中的各个连接边：
    基于各个关键点对应的所述类别信息，将每个关键点和与所述关键点所属的类别不同的其它关键点连接，形成所述关键点图模型中的连接边。
  7. 根据权利要求5所述的方法，其中，根据以下步骤生成所述关键点图模型中的各个连接边：
    基于各个关键点对应的所述类别信息、和预设的不同类别之间的匹配关系，将每个关键点和与所述关键点所属类别匹配的目标类别对应的关键点连接，形成所述关键点图模型中的连接边。
  8. 根据权利要求1至7任一所述的方法，其中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
    基于每个目标对象对应的各个关键点的信息，确定所述目标对象的行为类型。
  9. 根据权利要求1至7任一所述的方法，其中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
    基于每个目标对象对应的各个关键点的信息，生成针对所述目标对象的特效信息。
  10. 一种关键点检测装置，包括：
    获取模块，配置为获取待检测图像；
    第一生成模块，配置为基于所述待检测图像，生成图像特征图和多个关键点热图；所述图像特征图用于表征所述待检测图像中各个目标对象之间的相对位置关系；每个所述关键点热图中包含所述待检测图像的一种类别的关键点，不同类别的关键点对应所述目标对象的不同部位；
    第二生成模块，配置为基于所述图像特征图和多个所述关键点热图，生成关键点图模型；所述关键点图模型中包含所述待检测图像中不同类别的关键点的信息以及连接边的信息，每个连接边为两个不同类别的关键点之间的边；
    确定模块，配置为基于所述关键点图模型，确定属于同一目标对象的各个关键点。
  11. 根据权利要求10所述的装置，其中，所述确定模块，在基于所述关键点图模型，确定属于同一目标对象的各个关键点的情况下，配置为：
    基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度；
    基于确定的所述相关度，确定属于同一目标对象的各个关键点。
  12. 根据权利要求11所述的装置，其中，所述确定模块，在基于确定的所述相关度，确定属于同一目标对象的各个关键点的情况下，配置为：
    将对应的所述相关度大于设定阈值的各个关键点作为同一目标对象的关键点。
  13. 根据权利要求11或12所述的装置，其中，所述确定模块，在基于所述关键点图模型中各个关键点的信息以及所述连接边的信息，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度的情况下，配置为：
    针对每个关键点，基于所述关键点的信息，和所述关键点图模型中与所述关键点存在连接关系的其它关键点的信息，确定所述关键点的融合特征；
    基于各个关键点分别对应的融合特征，确定所述关键点图模型中存在连接关系的两个关键点之间的相关度。
  14. 根据权利要求10至13任一所述的装置，其中，所述关键点的信息包括位置信 息、类别信息、以及像素特征信息；
    所述第二生成模块，配置为根据以下步骤确定所述关键点图模型中各个关键点的信息：
    基于所述关键点热图，确定各个关键点的位置信息；
    基于每个所述关键点的位置信息，从所述图像特征图中提取所述关键点的像素特征信息，并基于所述关键点所属关键点热图的类别标签，确定所述关键点对应的类别信息。
  15. 根据权利要求14所述的装置，其中，所述第二生成模块，配置为根据以下步骤生成所述关键点图模型中的各个连接边：
    基于各个关键点对应的所述类别信息，将每个关键点和与所述关键点所属的类别不同的其它关键点连接，形成所述关键点图模型中的连接边。
  16. 根据权利要求14所述的装置，其中，所述第二生成模块，配置为根据以下步骤生成所述关键点图模型中的各个连接边：
    基于各个关键点对应的所述类别信息、和预设的不同类别之间的匹配关系，将每个关键点和与所述关键点所属类别匹配的目标类别对应的关键点连接，形成所述关键点图模型中的连接边。
  17. 根据权利要求10至16任一所述的装置，其中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
    行为类型确定模块，配置为基于每个目标对象对应的各个关键点的信息，确定所述目标对象的行为类型。
  18. 根据权利要求10至16任一所述的装置，其中，在基于所述关键点图模型，确定属于同一目标对象的各个关键点之后，还包括：
    特效信息生成模块，配置为基于每个目标对象对应的各个关键点的信息，生成针对所述目标对象的特效信息。
  19. 一种电子设备，包括：处理器、存储器和总线，所述存储器存储有所述处理器可执行的机器可读指令，当电子设备运行时，所述处理器与所述存储器之间通过总线通信，所述机器可读指令被所述处理器执行时执行如权利要求1至9任一所述的关键点检测方法的步骤。
  20. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行如权利要求1至9任一所述的关键点检测方法的步骤。
  21. 一种计算机程序产品，包括计算机可读代码，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行用于实现权利要求1至9任一项所述的关键点检测方法的步骤。
PCT/CN2021/075128 2020-06-30 2021-02-03 关键点检测方法、装置、电子设备及存储介质 WO2022001106A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2021539648A JP2022542199A (ja) 2020-06-30 2021-02-03 キーポイントの検出方法、装置、電子機器および記憶媒体
KR1020217021260A KR20220004009A (ko) 2020-06-30 2021-02-03 키 포인트 검출 방법, 장치, 전자 기기 및 저장 매체

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010622132.3 2020-06-30
CN202010622132.3A CN111783882B (zh) 2020-06-30 2020-06-30 关键点检测方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022001106A1 true WO2022001106A1 (zh) 2022-01-06

Family

ID=72761609

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/075128 WO2022001106A1 (zh) 2020-06-30 2021-02-03 关键点检测方法、装置、电子设备及存储介质

Country Status (5)

Country Link
JP (1) JP2022542199A (zh)
KR (1) KR20220004009A (zh)
CN (1) CN111783882B (zh)
TW (1) TWI774271B (zh)
WO (1) WO2022001106A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023221284A1 (zh) * 2022-05-19 2023-11-23 深圳大学 一种图关系网络人数统计方法及相关设备

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111783882B (zh) * 2020-06-30 2022-09-09 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质
CN111898642B (zh) * 2020-06-30 2021-08-13 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质
CN112336342B (zh) * 2020-10-29 2023-10-24 深圳市优必选科技股份有限公司 手部关键点检测方法、装置及终端设备
CN113762315A (zh) * 2021-02-04 2021-12-07 北京京东振世信息技术有限公司 图像检测方法、装置、电子设备和计算机可读介质
KR102660127B1 (ko) * 2023-04-06 2024-04-25 주식회사 써지컬에이아이 히트맵을 이용하여 엑스레이 영상에서 관절의 중심점을 검출하는 장치 및 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109948526A (zh) * 2019-03-18 2019-06-28 北京市商汤科技开发有限公司 图像处理方法及装置、检测设备及存储介质
US20190303677A1 (en) * 2018-03-30 2019-10-03 Naver Corporation System and method for training a convolutional neural network and classifying an action performed by a subject in a video using the trained convolutional neural network
CN110532984A (zh) * 2019-09-02 2019-12-03 北京旷视科技有限公司 关键点检测方法、手势识别方法、装置及系统
CN111783882A (zh) * 2020-06-30 2020-10-16 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质
CN111898642A (zh) * 2020-06-30 2020-11-06 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7657126B2 (en) * 2005-05-09 2010-02-02 Like.Com System and method for search portions of objects in images and features thereof
JP4935647B2 (ja) * 2007-11-29 2012-05-23 カシオ計算機株式会社 合成画像出力装置および合成画像出力処理プログラム
WO2013086429A2 (en) * 2011-12-09 2013-06-13 Veracyte, Inc. Methods and compositions for classification of samples
JP2015061577A (ja) * 2013-01-18 2015-04-02 株式会社東芝 動作情報処理装置
EP3494428A4 (en) * 2016-08-02 2020-04-08 Atlas5D, Inc. SYSTEMS AND METHODS FOR IDENTIFYING PEOPLE AND / OR IDENTIFYING AND QUANTIFYING PAIN, FATIGUE, MOOD AND INTENTION WITH PRIVACY PROTECTION
JP2019175321A (ja) * 2018-03-29 2019-10-10 大日本印刷株式会社 画像評価装置、画像評価方法及びコンピュータプログラム
CN108520251A (zh) * 2018-04-20 2018-09-11 北京市商汤科技开发有限公司 关键点检测方法及装置、电子设备和存储介质
TW202347098A (zh) * 2018-08-07 2023-12-01 李文傑 具有廣泛使用性的三維圖形使用者介面的系統及方法與對應的可讀式媒體
US11238612B2 (en) * 2018-08-28 2022-02-01 Beijing Jingdong Shangke Information Technology Co., Ltd. Device and method of tracking poses of multiple objects based on single-object pose estimator
JP7096175B2 (ja) * 2019-01-23 2022-07-05 Kddi株式会社 オブジェクト抽出方法および装置
US10643085B1 (en) * 2019-01-30 2020-05-05 StradVision, Inc. Method and device for estimating height and weight of passengers using body part length and face information based on human's status recognition
CN110322702B (zh) * 2019-07-08 2020-08-14 中原工学院 一种基于双目立体视觉系统的车辆智能测速方法
CN111339903B (zh) * 2020-02-21 2022-02-08 河北工业大学 一种多人人体姿态估计方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190303677A1 (en) * 2018-03-30 2019-10-03 Naver Corporation System and method for training a convolutional neural network and classifying an action performed by a subject in a video using the trained convolutional neural network
CN109948526A (zh) * 2019-03-18 2019-06-28 北京市商汤科技开发有限公司 图像处理方法及装置、检测设备及存储介质
CN110532984A (zh) * 2019-09-02 2019-12-03 北京旷视科技有限公司 关键点检测方法、手势识别方法、装置及系统
CN111783882A (zh) * 2020-06-30 2020-10-16 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质
CN111898642A (zh) * 2020-06-30 2020-11-06 北京市商汤科技开发有限公司 关键点检测方法、装置、电子设备及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023221284A1 (zh) * 2022-05-19 2023-11-23 深圳大学 一种图关系网络人数统计方法及相关设备

Also Published As

Publication number Publication date
CN111783882B (zh) 2022-09-09
TWI774271B (zh) 2022-08-11
JP2022542199A (ja) 2022-09-30
KR20220004009A (ko) 2022-01-11
TW202203211A (zh) 2022-01-16
CN111783882A (zh) 2020-10-16

Similar Documents

Publication Publication Date Title
WO2022001106A1 (zh) 关键点检测方法、装置、电子设备及存储介质
CN109657631B (zh) 人体姿态识别方法及装置
CN106203242B (zh) 一种相似图像识别方法及设备
WO2022001123A1 (zh) 关键点检测方法、装置、电子设备及存储介质
WO2020164270A1 (zh) 基于深度学习的行人检测方法、系统、装置及存储介质
WO2018228218A1 (zh) 身份识别方法、计算设备及存储介质
WO2020211339A1 (zh) 指静脉识别方法、装置、计算机设备及存储介质
Lin et al. Learning correspondence structures for person re-identification
WO2016107482A1 (zh) 确定人脸图像中人脸的身份标识的方法、装置和终端
WO2019033571A1 (zh) 面部特征点检测方法、装置及存储介质
JP2019109709A (ja) 画像処理装置、画像処理方法およびプログラム
JP7106742B2 (ja) 顔認識方法、装置、電子機器及びコンピュータ不揮発性読み取り可能な記憶媒体
KR20130095727A (ko) 비디오 내 객체들의 시맨틱 파싱
CN111401219B (zh) 一种手掌关键点检测方法和装置
CN111859002B (zh) 兴趣点名称生成方法及装置、电子设备和介质
CN111507285A (zh) 人脸属性识别方法、装置、计算机设备和存储介质
TWI776566B (zh) 動作識別方法、電腦設備及電腦可讀儲存介質
KR20220042335A (ko) 자동 수어 인식 방법 및 시스템
CN111401318A (zh) 动作识别方法及装置
CN113780145A (zh) 精子形态检测方法、装置、计算机设备和存储介质
CN118038303A (zh) 身份识别图像处理方法、装置、计算机设备和存储介质
US20240303848A1 (en) Electronic device and method for determining human height using neural networks
JP2016045884A (ja) パターン認識装置およびパターン認識方法
CN117854156A (zh) 一种特征提取模型的训练方法和相关装置
Zhang et al. Multiview facial landmark localization in RGB-D images via hierarchical regression with binary patterns

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021539648

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21834362

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21834362

Country of ref document: EP

Kind code of ref document: A1