CN118411764B - Dynamic bone recognition method, system, storage medium and electronic equipment - Google Patents
Dynamic bone recognition method, system, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN118411764B CN118411764B CN202410876073.0A CN202410876073A CN118411764B CN 118411764 B CN118411764 B CN 118411764B CN 202410876073 A CN202410876073 A CN 202410876073A CN 118411764 B CN118411764 B CN 118411764B
- Authority
- CN
- China
- Prior art keywords
- data
- bone
- preset
- determining
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000000988 bone and bone Anatomy 0.000 title claims abstract description 249
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012937 correction Methods 0.000 claims abstract description 49
- 230000004927 fusion Effects 0.000 claims abstract description 47
- 230000009471 action Effects 0.000 claims abstract description 32
- 238000007781 pre-processing Methods 0.000 claims abstract description 32
- 238000013136 deep learning model Methods 0.000 claims abstract description 24
- 238000013528 artificial neural network Methods 0.000 claims abstract description 21
- 210000003625 skull Anatomy 0.000 claims description 22
- 230000033001 locomotion Effects 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000005477 standard model Effects 0.000 claims description 5
- 230000002146 bilateral effect Effects 0.000 claims description 4
- 230000000295 complement effect Effects 0.000 claims description 4
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a dynamic bone recognition method, a system, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring multi-modal data through preset sensing equipment and respectively preprocessing according to the multi-modal data types to obtain multi-modal preprocessed data; determining fusion data according to the multi-mode preprocessing data through a preset strategy and determining key skeleton point data according to the fusion data through a preset deep learning model; determining correction data according to the key bone point data through a preset algorithm, determining correction fusion data according to the correction data and the multi-mode preprocessing data through a preset strategy, and determining correction key bone point data according to the correction fusion data through a preset deep learning model; and determining action characteristic data through a preset neural network according to the corrected key bone point data and determining a user bone model according to the corrected key bone point data and the action characteristic data. The invention solves the problem that a dynamic bone recognition method with high accuracy and robustness is lacking in the prior art.
Description
Technical Field
The invention relates to the field of computer vision and man-machine interaction, in particular to a dynamic bone recognition method, a system, a storage medium and electronic equipment.
Background
The action recognition refers to analyzing given action sequence data, recognizing and judging action categories contained in the given action sequence data, and is widely applied to the fields of video monitoring, man-machine interaction, augmented reality, automatic driving and the like. With the rapid development of 3D motion capture systems and advanced real-time 2D/3D pose estimation algorithms, bone-based motion recognition is increasingly receiving industry and academia attention.
The essence of the existing human behavior recognition is that the action characteristics of a human body are extracted from videos, images or skeleton sequences, and corresponding classification recognition is performed to further identify specific action types, wherein the skeleton sequence data has the characteristic of strong background interference resistance, and meanwhile, the data are sparse and easy to acquire, so that deep application is achieved.
The existing bone action recognition method still has the problem of low recognition precision, so that how to improve the accuracy and the robustness of bone action recognition becomes a technical problem to be solved urgently by the person skilled in the art.
Disclosure of Invention
Based on the above, the invention aims to provide a dynamic bone recognition method, a system, a storage medium and electronic equipment, which aim to solve the problem that a dynamic bone recognition method with high accuracy and robustness is lacked in the prior art.
A dynamic bone recognition method according to an embodiment of the present invention, the method comprising:
acquiring multi-modal data through preset sensing equipment, and respectively preprocessing according to the multi-modal data types to obtain multi-modal preprocessed data;
Determining fusion data according to the multi-mode preprocessing data through a preset strategy, and determining key skeleton point data according to the fusion data through a preset deep learning model;
Determining correction data according to the key bone point data through a preset algorithm, determining correction fusion data according to the correction data and the multi-mode preprocessing data through the preset strategy, and determining correction key bone point data according to the correction fusion data through the preset deep learning model;
And determining action characteristic data through a preset neural network according to the corrected key bone point data, and determining a user bone model according to the corrected key bone point data and the action characteristic data.
In addition, the dynamic bone recognition method according to the above embodiment of the present invention may further have the following additional technical features:
further, the step of determining the fusion data according to the multi-mode preprocessing data through a preset strategy includes:
Determining characteristic data corresponding to each mode through a preset convolutional neural network according to the multi-mode preprocessing data;
And determining fusion data according to the characteristic data through a preset vector conversion method and a weight matrix.
Further, the step of determining correction data according to the key bone point data through a preset algorithm includes:
According to the key bone point data, comparing the key bone point data with a preset database, and determining a plurality of bone model data with similarity to the key bone point data being larger than a preset value;
Integrating a plurality of bone model data to obtain calibrated bone model data;
And determining deviation bone point data according to the key bone point data and the correction bone model data through a preset method, and determining correction data according to the deviation bone point data through the preset deep learning model.
Further, the step of determining motion feature data through a preset neural network according to the corrected key skeleton point data comprises the following steps:
determining bone map data through a preset map neural network according to the corrected key bone point data and determining joint data according to the bone map data;
Determining dependency relationship data between each joint and adjacent joints through the preset graph neural network according to the joint data;
And respectively determining movement characteristic data of each joint according to the joint data and the dependency relationship data, wherein the movement characteristic data are combined to form the action characteristic data.
Further, the step of determining a plurality of bone model data having a similarity to the key bone point data greater than a predetermined value according to the comparison of the key bone point data with a predetermined database comprises:
determining backbone bone data and branch bone data according to the key bone point data, wherein the backbone bone data at least comprises skull data and trunk data;
comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value;
and comparing the branch bone data with a plurality of first bone model data to determine a plurality of second bone model data with deviation less than a second preset threshold, wherein the second preset threshold is less than the first preset threshold.
Further, the step of comparing the skull data and the torso data with a preset database to determine a plurality of first bone model data having a degree of deviation less than a first preset threshold value comprises, prior to:
classifying the main bone data and the branch bone data to determine regional bone data types, and comparing the regional bone data with the regional bone data types of standard models in the preset database to judge whether the types of the regional bone data types are missing;
If not, continuing to execute the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value;
if yes, determining full skeleton data according to the main skeleton data and the branch skeleton data through a preset rule, and continuously executing the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first skeleton model data with deviation degree smaller than a first preset threshold value until deviation skeleton point data are obtained;
And determining the correction data through the preset deep learning model according to the complement bone data and the deviation bone point data.
Further, the multi-modal data at least includes depth data, infrared data and RGB image data, and the steps of preprocessing according to the multi-modal data types to obtain multi-modal preprocessed data include:
Denoising the depth data through a bilateral filter;
enhancing the contrast of the infrared data through histogram equalization;
And carrying out normalization processing on the RGB image.
It is another object of an embodiment of the present invention to provide a dynamic bone recognition system, the system comprising:
The data processing module is used for acquiring multi-mode data through preset sensing equipment and respectively preprocessing the multi-mode data according to the multi-mode data types to obtain multi-mode preprocessed data;
the bone point data determining module is used for determining fusion data according to the multi-mode preprocessing data through a preset strategy and determining key bone point data according to the fusion data through a preset deep learning model;
The corrected bone point data determining module is used for determining corrected data according to the key bone point data through a preset algorithm, determining corrected fusion data according to the corrected data and the multi-mode preprocessing data through the preset strategy, and determining corrected key bone point data according to the corrected fusion data through the preset deep learning model;
And the bone model determining module is used for determining action characteristic data through a preset neural network according to the corrected key bone point data and determining a user bone model according to the corrected key bone point data and the action characteristic data.
It is a further object of an embodiment of the present invention to provide a storage medium having stored thereon a computer program which when executed by a processor performs the steps of the dynamic bone recognition method described above.
It is a further object of an embodiment of the present invention to provide an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, which processor implements the steps of the dynamic bone identification method described above when executing the program.
According to the invention, the data of different modes are obtained through the preset sensing equipment, and the multi-mode data are fused, so that the accuracy and the robustness of bone identification under different environments are improved. In addition, key skeleton point data are determined according to the fusion data, the key skeleton point data are compared with a preset database, correction data are determined through a preset algorithm, the correction fusion data are obtained through fusion of the correction data and the multi-mode data, and further data optimization is achieved through data feedback and correction and secondary fusion, so that accuracy of the key skeleton point data is improved. In addition, the dependency relationship among the joints is determined through the preset neural network, so that the action characteristics of the joints are determined by the bone point data of the joints and the bone point data of the adjacent joints, the quantity processing amount is reduced, the action recognition speed is improved, the accuracy of the action recognition is ensured through the dependency relationship, and the recognition efficiency and accuracy of complex actions can be improved. The accuracy and the robustness of the method are further improved. Therefore, the invention solves the problem that a dynamic bone recognition method with high accuracy and robustness is lacked in the prior art.
Drawings
FIG. 1 is a flow chart of a dynamic bone recognition method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of the results of a dynamic bone recognition system according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention;
The invention will be further described in the following detailed description in conjunction with the above-described figures.
Detailed Description
In order that the invention may be readily understood, a more complete description of the invention will be rendered by reference to the appended drawings. Several embodiments of the invention are presented in the figures. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
It will be understood that when an element is referred to as being "mounted" on another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
Example 1
Referring to fig. 1, a dynamic bone recognition method according to a first embodiment of the present invention is shown, and the method specifically includes steps S01-S04.
S01, acquiring multi-mode data through preset sensing equipment, and respectively preprocessing according to the multi-mode data types to obtain multi-mode preprocessed data;
Specifically, the multi-mode data at least comprises depth data, infrared data and RGB image data, and the depth data is denoised through a bilateral filter; enhancing the contrast of the infrared data through histogram equalization; and carrying out normalization processing on the RGB image. In the specific implementation, corresponding preprocessing modes are respectively carried out on the data of different modes, so that the accuracy of the data can be further improved, and the interference of invalid data is avoided.
S02, determining fusion data according to the multi-mode preprocessing data through a preset strategy, and determining key skeleton point data according to the fusion data through a preset deep learning model;
Specifically, determining characteristic data corresponding to each mode through a preset convolutional neural network according to the multi-mode preprocessing data; and determining fusion data according to the characteristic data through a preset vector conversion method and a weight matrix. After the data preprocessing is completed, feature extraction is carried out on the data of each mode through a convolutional neural network, features of different modes are integrated through connection fusion or feature level fusion, and fusion data with the same data type dimension as that required by a preset deep learning model is finally determined through vector conversion and a weight matrix. And determining key bone point data through a preset deep learning model. The method and the device realize effective mutual assistance of the data of each mode, and improve the identification accuracy and robustness under different environments.
S03, determining correction data through a preset algorithm according to the key bone point data, determining correction fusion data through the preset strategy according to the correction data and the multi-mode preprocessing data, and determining correction key bone point data through the preset deep learning model according to the correction fusion data;
Specifically, according to the comparison of the key bone point data and a preset database, determining a plurality of bone model data with similarity to the key bone point data larger than a preset value; integrating a plurality of bone model data to obtain calibrated bone model data; and determining deviation bone point data according to the key bone point data and the correction bone model data through a preset method, and determining correction data according to the deviation bone point data through the preset deep learning model. The key bone point data are compared with the preset database, so that the key bone point data are corrected, the situation that the virtual model is strange due to the fact that the bone data are wrong and the inverse joint or bone distribution is malformed is avoided, the correction data are determined, and the correction data are subjected to secondary fusion correction with the preprocessed multi-mode data, so that the accuracy of bone identification is further improved.
More specifically, determining backbone bone data and branch bone data according to the key bone point data, wherein the backbone bone data at least comprises skull data and trunk data; comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value; and comparing the branch bone data with a plurality of first bone model data to determine a plurality of second bone model data with deviation less than a second preset threshold, wherein the second preset threshold is less than the first preset threshold. When specific comparison is carried out, as the skull and the trunk bones of the human are easier to identify, the change amplitude of the bone morphology of the skull and the trunk bones is relatively smaller even if the human moves, and the branch bone data are similar under the condition that the skull and the trunk bones are similar, namely, the growth conditions of the bones of the limbs and other parts are generally similar under the condition that the body bones are consistent. Therefore, the data comparison amount is reduced by screening the main bone data and the preset database, the bone recognition efficiency is improved, and in addition, the first bone model data obtained by primarily screening the main bone data has high accuracy due to the strong correlation of the main bone data to the branch bone data. And because the form change amplitude of the backbone skeleton data is smaller, the screening difficulty is lower, and the screening speed and the accuracy can be ensured by setting a larger threshold value. Further, after screening the backbone skeleton data, the residual data amount is less, and secondary smaller comparison threshold screening is performed through the branch skeleton data, so that the required data can be rapidly and accurately screened out, the phenomenon of few individual differences is avoided, and the accuracy of skeleton recognition is improved.
Furthermore, the step of comparing the skull data and the torso data with a preset database to determine a plurality of first bone model data having a degree of deviation less than a first preset threshold further comprises, prior to: classifying the main bone data and the branch bone data to determine regional bone data types, and comparing the regional bone data with the regional bone data types of standard models in the preset database to judge whether the types of the regional bone data types are missing; if not, continuing to execute the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value; if yes, determining full skeleton data according to the main skeleton data and the branch skeleton data through a preset rule, and continuously executing the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first skeleton model data with deviation degree smaller than a first preset threshold value until deviation skeleton point data are obtained; and determining the correction data through the preset deep learning model according to the complement bone data and the deviation bone point data.
Since this skeleton recognition method is generally used for synchronizing with a model of a virtual world, it is mainly used for judging a person's motion by skeleton recognition to realize interaction of a person's model in the virtual world, and further the person's model in the virtual world is different from the physical appearance of a person in reality, and at least the motion is identical. Furthermore, since there is a disabled user's condition such as lack of arms, in order to ensure the user's experience, the limbs of the virtual character model should be normal, and thus it is also necessary to judge the kind of skeletal data. By way of example and not limitation, in some alternative embodiments, the bone data categories are divided into 206, i.e., corresponding to the number of human bones, but generally a coarser category distinction is made due to data processing efficiency issues. When the absence of the category is judged through the comparison of the category number, the bone data is complemented according to the existing main bone data, the branch bone data and standard model data in a database, and the complemented bone data is produced. And further, even disabled users are guaranteed, the modeling of the virtual model character is complete and limb actions are coordinated, so that the experience effect of the users is guaranteed.
S04, determining action feature data through a preset neural network according to the corrected key bone point data, and determining a user bone model according to the corrected key bone point data and the action feature data;
Specifically, bone map data are determined according to the corrected key bone point data through a preset map neural network, and joint data are determined according to the bone map data; determining dependency relationship data between each joint and adjacent joints through the preset graph neural network according to the joint data; and respectively determining movement characteristic data of each joint according to the joint data and the dependency relationship data, wherein the movement characteristic data are combined to form the action characteristic data.
The preset neural network transmits information among nodes through a message transmission mechanism, so that the state of each node can be updated according to the neighbor nodes of the node, and the dependency relationship among the nodes is learned. The learning of such dependencies allows the model to accurately predict motion even in the case of partial joint occlusion, as the model can infer the state of invisible joints from visible joints. In addition, the local information processing architecture of the preset neural network enhances the processing efficiency, because each node only processes the node information directly connected with the node, the speed and the accuracy of motion recognition are obviously improved, and the overall recognition efficiency is improved through the efficient graph data structure.
In summary, according to the dynamic bone recognition method in the embodiment of the invention, the data of different modes are obtained through the preset sensing equipment, and the multi-mode data are fused, so that the accuracy and the robustness of bone recognition under different environments are improved. In addition, key skeleton point data are determined according to the fusion data, the key skeleton point data are compared with a preset database, correction data are determined through a preset algorithm, the correction fusion data are obtained through fusion of the correction data and the multi-mode data, and further data optimization is achieved through data feedback and correction and secondary fusion, so that accuracy of the key skeleton point data is improved. In addition, the dependency relationship among the joints is determined through the preset neural network, so that the action characteristics of the joints are determined by the bone point data of the joints and the bone point data of the adjacent joints, the quantity processing amount is reduced, the action recognition speed is improved, the accuracy of the action recognition is ensured through the dependency relationship, and the recognition efficiency and accuracy of complex actions can be improved. The accuracy and the robustness of the method are further improved. Therefore, the invention solves the problem that a dynamic bone recognition method with high accuracy and robustness is lacked in the prior art.
Example two
Referring to fig. 2, a block diagram of a dynamic bone recognition system according to a second embodiment of the present invention is shown, the dynamic bone recognition system 200 includes: a data processing module 21, a bone point data determination module 22, a corrected bone point data determination module 23, and a bone model determination module 24, wherein:
The data processing module 21 acquires multi-mode data through preset sensing equipment and respectively performs preprocessing according to the multi-mode data types to obtain multi-mode preprocessed data;
the bone point data determining module 22 is configured to determine fusion data according to the multi-mode preprocessing data through a preset strategy, and determine key bone point data according to the fusion data through a preset deep learning model;
The corrected bone point data determining module 23 is configured to determine corrected data according to the key bone point data by using a preset algorithm, determine corrected fusion data according to the corrected data and the multi-mode preprocessing data by using the preset strategy, and determine corrected key bone point data according to the corrected fusion data by using the preset deep learning model;
the bone model determining module 24 is configured to determine motion feature data according to the corrected key bone point data through a preset neural network, and determine a user bone model according to the corrected key bone point data and the motion feature data.
Further, in other embodiments of the present invention, the multi-modal data includes at least depth data, infrared data, and RGB image data, and the data processing module 21 includes:
and the data preprocessing unit is used for denoising the depth data through a bilateral filter, enhancing the contrast of infrared data through histogram equalization and carrying out normalization processing on the RGB image.
The characteristic data determining unit is used for determining characteristic data corresponding to each mode through a preset convolutional neural network according to the multi-mode preprocessing data;
And the fusion data determining unit is used for determining fusion data according to the characteristic data through a preset vector conversion method and a weight matrix.
Further, in other embodiments of the present invention, the corrected bone point data determining module 23 includes:
the bone model data determining unit is used for determining a plurality of bone model data with similarity to the key bone point data being larger than a preset value according to comparison between the key bone point data and a preset database;
the correction bone model data determining unit is used for integrating a plurality of bone model data to obtain correction bone model data;
And the correction data determining unit is used for determining deviation bone point data according to the key bone point data and the correction bone model data through a preset method and determining correction data according to the deviation bone point data through the preset deep learning model.
Further, in other embodiments of the present invention, the bone model determination module 24 includes:
The joint data determining unit is used for determining skeleton map data through a preset map neural network according to the corrected key skeleton point data and determining joint data according to the skeleton map data;
The dependency relationship data determining unit is used for determining dependency relationship data between each joint and the adjacent joint through the preset graph neural network according to the joint data;
And the motion characteristic data determining unit is used for respectively determining motion characteristic data of each joint according to the joint data and the dependency relationship data, and the motion characteristic data are combined to form the motion characteristic data.
Further, in other embodiments of the present invention, the bone model data determining unit includes:
A skeleton point data classification subunit, configured to determine, according to the key skeleton point data, main skeleton data and branch skeleton data, where the main skeleton data includes at least skull data and trunk data;
A first bone model data determining subunit configured to compare the skull data and the torso data with a preset database to determine a plurality of first bone model data having a degree of deviation less than a first preset threshold;
A second bone model data determining subunit, configured to compare the branch bone data with a plurality of first bone model data to determine a plurality of second bone model data with a deviation less than a second preset threshold, where the second preset threshold is greater than the first preset threshold.
The bone point data partitioning subunit classifies the main bone data and the branch bone data to determine regional bone data types, and compares the regional bone data with the regional bone data types of standard models in the preset database to judge whether the types of the regional bone data types are missing;
a normal path execution subunit, configured to, when the category deficiency does not exist in the regional bone data, continue to execute the step of comparing the skull data and the torso data with a preset database to determine a plurality of first bone model data with a deviation degree smaller than a first preset threshold;
An abnormal path execution subunit, configured to determine, when the region bone data does not have a category deficiency, full bone data according to the main bone data and the branch bone data through a preset rule, and continuously execute the steps of comparing the skull data and the torso data with a preset database to determine a plurality of first bone model data with a deviation degree smaller than a first preset threshold until deviation bone point data is obtained;
and the abnormal path correction data determining subunit is used for determining the correction data through the preset deep learning model according to the complement bone data and the deviation bone point data.
The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments, and are not described herein again.
Example III
In another aspect, referring to fig. 3, a schematic diagram of an electronic device according to a third embodiment of the present invention is provided, including a memory 20, a processor 10, and a computer program 30 stored in the memory and capable of running on the processor, where the processor 10 implements the dynamic bone recognition method as described above when executing the computer program 30.
The processor 10 may be, among other things, a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments for running program code or processing data stored in the memory 20, e.g. executing an access restriction program or the like.
The memory 20 includes at least one type of readable storage medium including flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 20 may in some embodiments be an internal storage unit of the electronic device, such as a hard disk of the electronic device. The memory 20 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk provided on the electronic device, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), etc. Further, the memory 20 may also include both internal storage units and external storage devices of the electronic device. The memory 20 may be used not only for storing application software of an electronic device and various types of data, but also for temporarily storing data that has been output or is to be output.
It should be noted that the structure shown in fig. 3 does not constitute a limitation of the electronic device, and in other embodiments the electronic device may comprise fewer or more components than shown, or may combine certain components, or may have a different arrangement of components.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the dynamic bone recognition method as described above.
Those of skill in the art will appreciate that the logic and/or steps represented in the flow diagrams or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (7)
1. A method of dynamic bone identification, the method comprising:
acquiring multi-modal data through preset sensing equipment, and respectively preprocessing according to the multi-modal data types to obtain multi-modal preprocessed data;
Determining fusion data according to the multi-mode preprocessing data through a preset strategy, and determining key skeleton point data according to the fusion data through a preset deep learning model;
Determining correction data according to the key bone point data through a preset algorithm, determining correction fusion data according to the correction data and the multi-mode preprocessing data through the preset strategy, and determining correction key bone point data according to the correction fusion data through the preset deep learning model;
determining action characteristic data through a preset neural network according to the corrected key bone point data, and determining a user bone model according to the corrected key bone point data and the action characteristic data;
the step of determining correction data according to the key bone point data through a preset algorithm comprises the following steps:
Determining a plurality of bone model data similar to the key bone point data according to the comparison of the key bone point data and a preset database;
Integrating a plurality of bone model data to obtain calibrated bone model data;
Determining deviation bone point data through a preset method according to the key bone point data and the correction bone model data, and determining correction data through the preset deep learning model according to the deviation bone point data;
The step of determining a plurality of bone model data similar to the key bone point data according to the comparison of the key bone point data with a preset database comprises the following steps:
determining backbone bone data and branch bone data according to the key bone point data, wherein the backbone bone data at least comprises skull data and trunk data;
comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value;
comparing the branch bone data with a plurality of first bone model data to determine a plurality of second bone model data with deviation less than a second preset threshold, wherein the second preset threshold is greater than the first preset threshold;
the step of comparing the skull data and the torso data with a preset database to determine a plurality of first bone model data having a degree of deviation less than a first preset threshold value may be preceded by:
classifying the main bone data and the branch bone data to determine regional bone data types, and comparing the regional bone data with the regional bone data types of standard models in the preset database to judge whether the types of the regional bone data types are missing;
If not, continuing to execute the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first bone model data with deviation degree smaller than a first preset threshold value;
if yes, determining full skeleton data according to the main skeleton data and the branch skeleton data through a preset rule, and continuously executing the steps of comparing the skull data and the trunk data with a preset database to determine a plurality of first skeleton model data with deviation degree smaller than a first preset threshold value until deviation skeleton point data are obtained;
And determining the correction data through the preset deep learning model according to the complement bone data and the deviation bone point data.
2. The method of claim 1, wherein determining fusion data from the multi-modal pre-processing data via a preset strategy comprises:
Determining characteristic data corresponding to each mode through a preset convolutional neural network according to the multi-mode preprocessing data;
And determining fusion data according to the characteristic data through a preset vector conversion method and a weight matrix.
3. The dynamic bone recognition method according to claim 1, wherein the step of determining motion feature data through a preset neural network from the corrected key bone point data comprises:
determining bone map data through a preset map neural network according to the corrected key bone point data and determining joint data according to the bone map data;
Determining dependency relationship data between each joint and adjacent joints through the preset graph neural network according to the joint data;
And respectively determining movement characteristic data of each joint according to the joint data and the dependency relationship data, wherein the movement characteristic data are combined to form the action characteristic data.
4. The method according to claim 1, wherein the multi-modal data includes at least depth data, infrared data, and RGB image data, and the step of preprocessing the multi-modal data according to the multi-modal data type to obtain multi-modal preprocessed data includes:
Denoising the depth data through a bilateral filter;
enhancing the contrast of the infrared data through histogram equalization;
And carrying out normalization processing on the RGB image.
5. A dynamic bone recognition system for implementing the dynamic bone recognition method according to any one of claims 1 to 4, the system comprising:
The data processing module is used for acquiring multi-mode data through preset sensing equipment and respectively preprocessing the multi-mode data according to the multi-mode data types to obtain multi-mode preprocessed data;
the bone point data determining module is used for determining fusion data according to the multi-mode preprocessing data through a preset strategy and determining key bone point data according to the fusion data through a preset deep learning model;
The corrected bone point data determining module is used for determining corrected data according to the key bone point data through a preset algorithm, determining corrected fusion data according to the corrected data and the multi-mode preprocessing data through the preset strategy, and determining corrected key bone point data according to the corrected fusion data through the preset deep learning model;
And the bone model determining module is used for determining action characteristic data through a preset neural network according to the corrected key bone point data and determining a user bone model according to the corrected key bone point data and the action characteristic data.
6. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the dynamic bone recognition method according to any one of claims 1 to 4.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the dynamic bone identification method according to any one of claims 1-4 when the program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410876073.0A CN118411764B (en) | 2024-07-02 | 2024-07-02 | Dynamic bone recognition method, system, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410876073.0A CN118411764B (en) | 2024-07-02 | 2024-07-02 | Dynamic bone recognition method, system, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118411764A CN118411764A (en) | 2024-07-30 |
| CN118411764B true CN118411764B (en) | 2024-10-18 |
Family
ID=91990203
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410876073.0A Active CN118411764B (en) | 2024-07-02 | 2024-07-02 | Dynamic bone recognition method, system, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118411764B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112287759A (en) * | 2020-09-26 | 2021-01-29 | 浙江汉德瑞智能科技有限公司 | Tumble detection method based on key points |
| CN114782992A (en) * | 2022-04-29 | 2022-07-22 | 常州大学 | Super-joint and multi-mode network and behavior identification method thereof |
| CN117456558A (en) * | 2023-11-09 | 2024-01-26 | 四川大学 | Human body posture estimation and control method based on camera and related equipment |
| CN117831120A (en) * | 2023-12-14 | 2024-04-05 | 南京大学 | Motion recognition system and method based on human skeleton point motion characteristics |
Family Cites Families (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180053309A1 (en) * | 2016-08-22 | 2018-02-22 | Seiko Epson Corporation | Spatial Alignment of M-Tracer and 3-D Human Model For Golf Swing Analysis Using Skeleton |
| CN106981075A (en) * | 2017-05-31 | 2017-07-25 | 江西制造职业技术学院 | The skeleton point parameter acquisition devices of apery motion mimicry and its recognition methods |
| CN110472612B (en) * | 2019-08-22 | 2021-09-07 | 海信集团有限公司 | Human behavior recognition method and electronic equipment |
| CN111437583B (en) * | 2020-04-10 | 2021-06-29 | 哈尔滨工业大学 | Badminton basic action auxiliary training system based on Kinect |
| CN111476181B (en) * | 2020-04-13 | 2022-03-04 | 河北工业大学 | Human skeleton action recognition method |
| CN112200819A (en) * | 2020-10-16 | 2021-01-08 | 河海大学 | Human body three-dimensional skeleton model marking method based on data driving and template guiding |
| CN114694243A (en) * | 2020-12-28 | 2022-07-01 | 普天信息技术有限公司 | Fall detection method and device, electronic equipment and storage medium |
| CN112801061A (en) * | 2021-04-07 | 2021-05-14 | 南京百伦斯智能科技有限公司 | Posture recognition method and system |
| CN113408455B (en) * | 2021-06-29 | 2022-11-29 | 山东大学 | Action identification method, system and storage medium based on multi-stream information enhanced graph convolution network |
| CN113469018B (en) * | 2021-06-29 | 2024-02-23 | 中北大学 | Multi-modal interactive behavior recognition method based on RGB and three-dimensional skeleton |
| CN113537148B (en) * | 2021-08-11 | 2024-04-19 | 泰康保险集团股份有限公司 | Human body action recognition method and device, readable storage medium and electronic equipment |
| CN113610969B (en) * | 2021-08-24 | 2024-03-08 | 国网浙江省电力有限公司双创中心 | Three-dimensional human body model generation method and device, electronic equipment and storage medium |
| JPWO2023079783A1 (en) * | 2021-11-05 | 2023-05-11 | ||
| CN113902995B (en) * | 2021-11-10 | 2024-04-02 | 中国科学技术大学 | Multi-mode human behavior recognition method and related equipment |
| JP2023141303A (en) * | 2022-03-23 | 2023-10-05 | 富士通株式会社 | Skeleton recognition program, skeleton recognition method and skeleton recognition device |
| CN114724241A (en) * | 2022-03-29 | 2022-07-08 | 平安科技（深圳）有限公司 | Motion recognition method, device, equipment and storage medium based on skeleton point distance |
| CN115761076A (en) * | 2022-11-25 | 2023-03-07 | 网易（杭州）网络有限公司 | Bone posture data, bone posture determination method, device, medium and equipment |
| CN116824689A (en) * | 2023-04-23 | 2023-09-29 | 汕头大学 | Bone sequence behavior recognition method, device, equipment and storage medium |
| CN117115917A (en) * | 2023-09-11 | 2023-11-24 | 云南师范大学 | Teacher behavior recognition method, device and medium based on multi-modal feature fusion |
| CN117197891A (en) * | 2023-09-11 | 2023-12-08 | 同济大学 | Multi-mode bone action recognition method and device |
| CN117612249A (en) * | 2023-11-13 | 2024-02-27 | 陕西彬长小庄矿业有限公司 | Underground miner dangerous behavior identification method and device based on improved OpenPose algorithm |
| CN117690163A (en) * | 2023-12-18 | 2024-03-12 | 广州大学 | 3D human skeleton key point data enhancement method and system |
| CN117854155B (en) * | 2024-03-07 | 2024-05-14 | 华东交通大学 | Human skeleton action recognition method and system |
| CN118247706A (en) * | 2024-03-29 | 2024-06-25 | 清华大学 | Three-dimensional attitude estimation method and device based on fine tuning standard model and storage medium |
-
2024
- 2024-07-02 CN CN202410876073.0A patent/CN118411764B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112287759A (en) * | 2020-09-26 | 2021-01-29 | 浙江汉德瑞智能科技有限公司 | Tumble detection method based on key points |
| CN114782992A (en) * | 2022-04-29 | 2022-07-22 | 常州大学 | Super-joint and multi-mode network and behavior identification method thereof |
| CN117456558A (en) * | 2023-11-09 | 2024-01-26 | 四川大学 | Human body posture estimation and control method based on camera and related equipment |
| CN117831120A (en) * | 2023-12-14 | 2024-04-05 | 南京大学 | Motion recognition system and method based on human skeleton point motion characteristics |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118411764A (en) | 2024-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110728330A (en) | Object identification method, device, equipment and storage medium based on artificial intelligence | |
| CN111754396B (en) | Face image processing method, device, computer equipment and storage medium | |
| US20130070997A1 (en) | Systems, methods, and media for on-line boosting of a classifier | |
| CN109859836B (en) | Medical image recognition method and device | |
| US11222069B2 (en) | Low-power calculation of a signature of a media unit | |
| CN111914665A (en) | Face shielding detection method, device, equipment and storage medium | |
| CN109583364A (en) | Image-recognizing method and equipment | |
| US11275971B2 (en) | Bootstrap unsupervised learning | |
| KR20210051473A (en) | Apparatus and method for recognizing video contents | |
| CN110781770A (en) | Living body detection method, device and equipment based on face recognition | |
| CN117036895A (en) | Multi-task environment sensing method based on point cloud fusion of camera and laser radar | |
| CN117854155B (en) | Human skeleton action recognition method and system | |
| CN118411764B (en) | Dynamic bone recognition method, system, storage medium and electronic equipment | |
| CN113610071B (en) | Face living body detection method and device, electronic equipment and storage medium | |
| CN112329736B (en) | Face recognition method and financial system | |
| CN112686851B (en) | Image detection method, device and storage medium | |
| KR101972095B1 (en) | Method and Apparatus of adding artificial object for improving performance in detecting object | |
| CN115830720A (en) | Living body detection method, living body detection device, computer equipment and storage medium | |
| CN113348463B (en) | Information processing method and device | |
| CN117636266B (en) | Method and system for detecting safety behaviors of workers, storage medium and electronic equipment | |
| CN117035125B (en) | Decision system, method and device of distributed artificial intelligence model | |
| CN117036377A (en) | Image processing method, image processing device, computer device, storage medium, and program product | |
| CN115937766A (en) | Identification method, identification device, electronic equipment and computer-readable storage medium | |
| CN117152440A (en) | Object recognition method and device based on image segmentation and electronic equipment | |
| CN115880742A (en) | Facial abnormal expression recognition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |