TWI638278B - An online verification method and system for real-time gesture detection - Google Patents

Abstract

The invention belongs to the field of electronic technology, and particularly relates to an online verification method for instant gesture detection, which comprises the following steps: Step 1: The image acquisition module captures images in a visual range in real time; Step 2, an embedded terminal passes loading training a good model performs gesture recognition and tracking monitoring on the captured image; step 3, records the recognition result and responds to the recognition result; and steps 4, analyzes the correctness of the recognition result and retrains according to the set rule to obtain a new one. Model to verify the accuracy of the new model. Step 5, updating the previously trained model with the new model. The above technical solution realizes a set of instant gesture recognition method and system, and provides a more accurate online model optimization system than the traditional non-depth camera based gesture recognition, which is beneficial to improve the accuracy of recognition.

Description

Online verification method and system for instant gesture detection

The invention belongs to the field of electronic technologies, and in particular relates to an online verification method and system for gesture detection.

With the maturity of embedded technology, various smart products have sprung up. In smart devices, machine vision has always been a hot topic of concern. At present, the existing gesture technologies are mainly divided into two categories: one is a three-dimensional visual recognition based on a depth camera. In addition to setting a camera, a depth camera is also provided, which can acquire spatial information through infrared reflection, enriching the features captured by the camera. The identification accuracy has been greatly increased, and it has been applied to front-end technology products, such as Microsoft's XBox series. Its supporting kinect device is a relatively mature depth camera in the industry, which can be used with games by gestures or body gestures. Interaction, however, the depth camera-based 3D visual recognition technology enhances the perception of the image, but is limited by the timeliness, stability and some architectural compatibility of the depth camera, and cannot be widely used. One type of technology is based on traditional two-dimensional imaging cameras, such as Haier's smart TV, the gesture image captured by the camera, and the control operation of the TV. The principle of its implementation is mainly based on loading the pre-trained model to filter out the match. A window that specifies the gesture. However, such traditional solutions have some problems: 1) different scenarios and environments are limited to pre-trained models; 2) accuracy and immediacy cannot be met at the same time.

The present invention provides an online verification method and system for instant gesture detection to solve the problems of the prior art;

The specific technical solution is as follows: an online verification method for instant gesture detection, which includes the following steps: Step 1: The image acquisition module instantly captures an image within a visual range; Step 2, an embedded terminal is trained by loading The model performs gesture recognition and tracking monitoring on the collected image; Step 3: recording the recognition result and responding to the recognition result; Step 4, analyzing the correctness of the recognition result and retraining to obtain a new model according to the set rule, Verifying the accuracy of the new model; Step 5, updating the previously trained model with the new model.

In the online verification method of the instant gesture detection, the step 4 is specifically as follows: Step 41: The recognition result is periodically uploaded to a background server, and the background server verifies the correctness of the recognition result by using a deep learning method; Step 42 Recording the error result that the recognition result is incorrect, determining that the error case reaches the set number or collecting the set time, adding the data of the error case to the training data of the previous model, and retraining to obtain new Model; Step 43, using a standard validation set to analyze the quality of the new model.

In the online verification method of the instant gesture detection, the step 5 is specifically as follows: Step 51: When determining that the new model is better than the previous model, the background server sends a request for upgrading the model to the embedded terminal; Step 52: The embedded terminal automatically downloads the new model to the embedded terminal in response to the request.

In the above-mentioned online verification method for instant gesture detection, the step 2 is specifically as follows: Step 21: When there is a moving object in the visual range, the embedded terminal starts gesture recognition; Step 22, loading the pre-trained model The target gesture is filtered out from the image, and the subsequent image is tracked and detected.

The invention also includes an online verification system for instant gesture detection, comprising: an image acquisition module for instantly capturing images in a visual range; a gesture recognition tracking module, located at an embedded terminal, and the image The acquisition module is connected, and the captured image is subjected to gesture recognition and tracking monitoring by loading the trained model; the recording response module is connected with the gesture recognition tracking module, and is used for recording the recognition result and responding to the Determining the result of the recognition; connecting the operation module to the record response module for analyzing the correctness of the recognition result and retraining the new model according to the set rule, and verifying the accuracy of the new model; And a module connected to the verification operation module, configured to update the model according to the new model.

In the above-mentioned online verification system for instant gesture detection, the verification operation module is located at a background server end, and includes: a detection back test sub-module connected to the record response module for returning the recognition result The error identification information and the noise information are recorded; the model training sub-module is connected to the detection back-test sub-module, and the wrong identification information and noise information of the set quantity or set time are added to the training data. Re-training to obtain a new model; verifying the sub-module, connecting with the model training sub-module, and performing quantitative evaluation on the new model according to the regularly updated verification set, when the new model is better than the previous model , issue a message to update the model.

The above-mentioned online verification system for instant gesture detection, the recording response module includes a visual feedback unit, and the visual feedback unit responds to the recognition result on a display interface of the embedded terminal by displaying a corresponding icon; and/ Or the recording response module includes an utterance feedback unit that responds to the recognition result by playing music or collecting music.

In the above-mentioned online verification system for instant gesture detection, the recording response module is located at the embedded terminal.

In the above-mentioned online verification system for instant gesture detection, the image acquisition module adopts a two-dimensional imaging camera.

Also included is an embedded smart device, including the online verification system for instant gesture detection described above.

The beneficial effects: the above technical solution realizes a set of instant gesture recognition method and system, and provides a more accurate online model optimization system than the traditional non-depth camera based gesture recognition, which is beneficial to improve the accuracy of recognition.

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be noted that the embodiments in the present invention and the features in the embodiments may be combined with each other without conflict.

The invention is further illustrated by the following figures and specific examples, but is not to be construed as limiting.

Referring to FIG. 1, an online verification method for instant gesture detection includes the following steps:

Step 1: The image acquisition module instantly captures an image within a visual range;

Step 2: an embedded terminal performs gesture recognition and tracking monitoring on the collected image by loading the trained model;

Step 3, recording the recognition result and responding to the recognition result;

Step 4: Analyze the correctness of the recognition result and retrain the new model according to the set rules to verify the accuracy of the new model;

Step 5, updating the previously trained model with the new model.

In the prior art, a large amount of data is collected in advance, and a partial image including a gesture is intercepted as a positive sample, and more negative sample images not including a gesture are intercepted as a training set. Then use the training set for the appropriate algorithm to train the model, load the pre-trained model before recognition, and calculate whether each image contains gestures, which is limited by the pre-trained model, for different scenes and Environment, affecting the effect of recognition, the invention records and analyzes the recognition result, and re-trains the new model according to specific data to update, and provides a more accurate online model optimization system, which is beneficial to improve the accuracy of recognition. And the target gestures appearing in any field of view can be effectively recognized.

The above online verification method for instant gesture detection, step 4 is as follows:

Step 41: The recognition result is periodically uploaded to a background server, and the background server uses the deep learning method to verify the correctness of the recognition result;

Step 42: Record an error case in which the recognition result is incorrect, determine that the error case reaches the set number or collect the set time, add the data of the error case to the training data of the previous model, and retrain to obtain the new model;

In step 43, a standard validation set is used to analyze the quality of the new model.

The background server accepts the recognition result sent by the front end and records it, and then uses the more accurate detection method to backtest the result, and records the wrong identification information and some noise information. At regular intervals, some detected data is added to the training data to train the model. After the training model is derived, the new model is quantitatively evaluated based on a set of regularly updated validation sets.

Most of the existing detection and recognition algorithms have problems in that the speed of the algorithm and the accuracy of the algorithm cannot be balanced. Accurate models generally require a large amount of computation, which is difficult to meet the requirements of real-time interactive systems, while fast algorithms are prone to problems of misidentification and low recall rate. The present invention sets the corresponding functions on the embedded terminal and the background server. The embedded terminal provides instant and fast recognition. For performance reasons, the detection operation is performed only when an object motion is detected, which greatly reduces the resource occupancy of the system and simultaneously tracks the detected area, which speeds up detection. The data reduces the system's resource consumption, and the system's timeliness and stability are greatly improved. In the background server, it provides more precise functions. Since the background server operation is regularly updated, the timeliness requirement is very low. The data transmitted to the background server through the embedded terminal is periodically used by the background server-side algorithm to verify the correct rate of recognition. And periodically use the collected data to retrain the client's detection model. In actual use, the client will be deployed to different environments, and in the initial use, different levels of false detection may occur. But after several rounds of inspection and retraining on the server side, the new model will be fully applicable to the deployed environment. Meet the dual guarantee of timeliness and accuracy.

The above online verification method for instant gesture detection, step 5 is as follows:

Step 51: When determining that the new model is better than the previous model, the background server sends a request for upgrading the model to the embedded terminal;

Step 52: The embedded terminal responds to the request, and the background server automatically downloads the new model to the embedded terminal.

After getting a trained new model, use a standard test set to analyze the quality of the model. At a specific time, if a new model is generated that is better than the embedded terminal, the background server sends an update request. When the embedded terminal responds to the request, the background server automatically downloads the new model to the embedded terminal. Each user can get a customized model so that the embedded terminal gesture recognition system can adapt to different environments.

The above online verification method for instant gesture detection, step 2 is as follows:

Step 21: when there is a moving object in the visual range, the embedded terminal starts gesture recognition;

In step 22, the pre-trained model is loaded, the target gesture is filtered out from the image, and the subsequent image is tracked and detected.

Specifically, the collected image is detected by the classifier after the training is implemented. If the target gesture appears, the response of the response is recorded and given, and the position of the gesture is recorded, and the subsequent image is tracked and detected.

Also included is an online verification system for instant gesture detection, wherein, referring to FIG. 5,

The image acquisition module 11 is configured to instantly capture an image in a visual range;

The gesture recognition tracking module 12 is located in an embedded terminal 1 and is connected to the image acquisition module 11 to perform gesture recognition and tracking monitoring on the captured image by loading the trained model.

The record response module 13 is connected to the gesture recognition tracking module 12 for recording the recognition result and responding to the recognition result;

The verification operation module 20 is connected to the record response module 13 for analyzing the correctness of the recognition result and retraining to obtain a new model according to the set rule, and verifying the accuracy of the new model;

The model update module 21 is connected to the verification operation module 20 for updating the previous model according to the new model.

The gesture recognition tracking module 12 can also be a gesture recognition program running on the embedded terminal 1 , which is equipped with an instant monitoring function, and provides monitoring results and data to a background server 2 for detecting back testing. The embedded terminal 1 can Independent of the background server 2 running independently without a network environment.

In the above-mentioned online verification system for instant gesture detection, the verification operation module 20 is located at a background server end 2, and includes:

Detecting the back test sub-module, connecting with the record response module, for performing backtesting on the recognition result, and recording the wrong identification information and noise information;

The model training sub-module is connected with the detection back-test sub-module, and adds the wrong identification information and noise information of the set quantity or set time to the training data, and re-trains to obtain a new model;

The verification sub-module is connected with the model training sub-module, and the new model is quantitatively evaluated according to the verification set of the timing update. When the new model is better than the previous model, the message of updating the model is issued.

The background server 2 can set the data collection and detection function, perform more accurate deep learning through the background server 2, perform backtesting analysis on the detection result of the embedded terminal 1, record the occurrence of the false detection, and time the training data. , update the new model to embedded terminal 1.

The above deep learning can be performed by constructing a multi-layered neural network, and the underlying convolution layer extracts basic information of the image, such as edge or point information. Then extract more abstract features layer by layer. For example, in the example of gestures, the middle layer extracts information such as skin color and skin folds. The higher network layer extracts the local features of the gesture, and finally fits the most connected layer. A reasonable classification function. The whole process is automatic training, although it takes a little time, but it is a background optimization update service, no need to worry about timeliness. At the same time, the background server can collect training data and retrain the model of the deep network at specific times. In order to ensure that the model accuracy of the background server 2 is higher than that of the embedded terminal 1, the purpose of verifying and optimizing the update can be achieved.

In the above-mentioned online verification system for instant gesture detection, the record response module 13 includes a visual feedback unit that responds to the recognition result by displaying a corresponding icon on the display interface of the embedded terminal; and/or

The recording response module includes an utterance feedback unit that responds to the recognition result by playing music or collecting music.

Other services of the embedded terminal 1. After receiving the instruction of the target gesture, an interactive reaction such as playing or stopping the music is made, and corresponding icons and visual effects are displayed on the external display module of the embedded terminal.

In the above-mentioned online verification system for instant gesture detection, the record response module 13 is located at the embedded terminal.

In the above-mentioned online verification system for instant gesture detection, the image acquisition module 11 can adopt a two-dimensional imaging camera for acquiring an instant image, and has a static image and a video capture function on 30 frames per second.

Also included is an embedded smart device, including the online verification system for instant gesture detection described above. The embedded smart device can be a robot running an embedded system.

A specific embodiment, referring to FIG. 6, an HD camera is connected to the embedded smart device through a MIPI (Mobile Industry Processor Interface) or a USB interface; the entire gesture control example is shown in FIG. 6:

In the embedded terminal: the HD camera instantly captures the image data appearing in the visual range. The gesture recognition system is activated only when there is a moving object in the camera range. When the target gesture is detected, it will be recorded immediately. The partial graphics area of the target gesture is then executed according to the different target gestures. For example, when a gesture for playing music appears, the local music interface is called to start playing music. If the recognized target gesture is a command to collect music, a favorite icon will appear on the screen, and the interface of the music collection will be called to add the music played at that time to the favorite list.

On the background server side: the recognition result recorded in the embedded terminal will be uploaded to the background server periodically, and the system will use the deep learning method to verify the correctness of the recognition result, and record the wrong case. When the wrong case reaches a certain amount or a certain amount of time is collected, the background program will add these error cases to the original training data and retrain the model. Once a new model is obtained, a standard validation set is used to analyze the quality of the new model. When the new model obtained is better than the original model, the server sends a request to upgrade the model to the embedded terminal. After the embedded terminal responds, the server automatically downloads the new model to the client. After multiple iterations, the accuracy of the recognition will be greatly improved.

The above is only a preferred embodiment of the present invention, and is not intended to limit the scope of the embodiments and the scope of the present invention, and those skilled in the art should be able to The resulting solutions to the obvious variations are intended to be included within the scope of the invention.

S1-S5‧‧‧ steps

41-43‧‧‧Steps

51-52‧‧‧Steps

21-22‧‧‧Steps

1‧‧‧ embedded terminal

2‧‧‧Backend server side

11‧‧‧Image Acquisition Module

12‧‧‧ gesture recognition tracking module

13‧‧‧Memory Impact Module

20‧‧‧Experience operation module

21‧‧‧Model Update Module

1 is a schematic flowchart of a method according to the present invention; FIG. 2 is a schematic flowchart of step 4 of the present invention; FIG. 3 is a schematic flowchart of step 5 of the present invention; FIG. 4 is a schematic flowchart of step 2 of the present invention; BRIEF DESCRIPTION OF THE DRAWINGS FIG. 6 is a schematic structural view of a specific embodiment of the present invention.

Claims (8)

An online verification method for instant gesture detection includes the following steps: Step 1: The image acquisition module instantly captures an image within a visual range; and Step 2, an embedded terminal loads the trained image by loading the trained model For example, performing gesture recognition and tracking monitoring; step 3, recording the recognition result and responding to the recognition result; and step 4, analyzing the correctness of the recognition result and retraining according to the set rule to obtain a new model, and verifying the accuracy of the new model And step 5, updating the previously trained model with the new model; wherein the step 4 is specifically as follows: Step 41, the recognition result is periodically uploaded to a background server, and the background server utilizes deep learning The method verifies the correctness of the recognition result; in step 42, records the error case that the recognition result is incorrect, determines that the error case reaches the set number, or collects the set time, and adds the data of the error case to the previous place In the training data of the model, retraining to obtain a new model; in step 43, using a standard validation set to analyze the new model Type of quality. The online verification method for instant gesture detection according to claim 1, wherein the step 5 is specifically as follows: Step 51: when the new model is judged to be superior to the previous model, the background server is embedded in the background The terminal sends a request to upgrade the model; in step 52, the embedded terminal automatically downloads the new model to the embedded terminal in response to the request. The online verification method for instant gesture detection according to the first aspect of the invention, wherein the step 2 is specifically as follows: Step 21: when there is a moving object in the visual range, the embedded terminal starts gesture recognition; In step 22, the pre-trained model is loaded, the target gesture is filtered out from the image, and subsequent images are tracked and detected. An online verification system for instant gesture detection, comprising: an image acquisition module for capturing an image in a visual range; the gesture recognition tracking module is located at an embedded terminal and connected to the image acquisition module. Performing gesture recognition and tracking monitoring on the captured image by loading the trained model; recording a response module connected to the gesture recognition tracking module for recording the recognition result and responding to the recognition result; An operation module is connected to the record response module for analyzing the correctness of the recognition result and retraining to obtain a new model according to the set rule, and verifying the accuracy of the new model; the model update module, and the The test operation module is connected to update the model according to the new model; wherein the verification operation module is located at a background server end, and includes: detecting a back test sub-module, and connecting with the record response module For performing backtesting on the recognition result, recording the wrong identification information and noise information; the model training sub-module, and the detecting back-testing sub-module Then, the error identification information and the noise information of the set quantity or the set time are added to the training data, and the new model is retrained; the verification sub-module is connected with the model training sub-module, and the verification set is updated according to the timing. The new model is quantitatively evaluated, and when the new model is better than the previous model, a message to update the model is issued. An online verification system for instant gesture detection as described in claim 4, wherein the recording response module includes a visual feedback unit, and the visual feedback unit passes Displaying a corresponding icon in response to the recognition result on a display interface of the embedded terminal; and/or the recording response module includes an utterance feedback unit, wherein the utterance feedback unit responds to the recognition by playing music or collecting music result. An online verification system for instant gesture detection as described in claim 4, wherein the recording response module is located in the embedded terminal. The online verification system for instant gesture detection according to claim 4, wherein the image acquisition module adopts a two-dimensional imaging camera. An embedded smart device employing an online verification system for instant gesture detection as described in claim 4 or 5 or 6 or 7.