CN111488823A - Augmented dimensional gesture recognition and interaction system and method based on 2D lidar - Google Patents

Abstract

The present invention proposes a two-dimensional laser radar-based augmented dimensional gesture recognition and interaction system and method, including a laser radar swing module, a three-dimensional point cloud reconstruction module, a graphic module, a gesture recognition and feature extraction module, a multi-frame comparison module and an interaction module. control module. The data is obtained by the swinging lidar, and the period information is obtained in the data of the 3D point cloud reconstruction module, so as to estimate the third dimensional data and obtain the 3D point cloud. Graph the point cloud of each cycle through the graphing module. In the gesture recognition and feature extraction module, the trained network is used for gesture recognition, and feature extraction is performed in the graphed graph. The multi-frame comparison module recognizes gestures with long and multi-gesture combinations. The interactive control module completes intention identification and interactive control. The invention has the advantages of simple structure, high precision and high cost performance. The interaction method of the present invention can be applied to various occasions and applications, and has high flexibility.

Description

Augmented dimensional gesture recognition and interaction system and method based on 2D lidar

technical field

The invention belongs to the fields of human-computer interaction and laser radar, and is applied to gesture recognition in demonstration and interaction to realize interaction, in particular to a two-dimensional laser radar-based augmented dimensional gesture recognition and interaction system and method.

Background technique

With the penetration of multimedia technology into people's lives, the development of AR and VR technology, the interactive technology applied to them is also rapidly changing and developing. Higher demands are placed on the reliability, ease of use, insensitivity and immersion of the use experience of interactive technology. Human-computer interaction technology is no longer limited to simple key operations such as keyboard and mouse. More advanced interaction technologies such as somatosensory interaction, gesture interaction, facial expression interaction, and gesture interaction have been proposed and applied, and even brain-computer interface has also been widely concerned. and research.

With the development of lidar technology, lidar has been widely used due to its high precision and good stability. The low-cost two-dimensional lidar makes the application of lidar technology in many fields without cost restrictions.

Interaction technology based on gesture recognition is a very promising technology in presentation and interaction. Among them, image-based methods are more popular. Compared with gesture recognition based on camera images, lidar can provide more accurate information such as position and angle, which provides the possibility for higher-precision interactive technology.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the purpose of the present invention is to provide a system and method for augmented dimensional gesture recognition and interaction based on two-dimensional laser radar. A swinging accessory is added to the two-dimensional lidar to realize the increased dimension of the radar, and gesture recognition and intention analysis are realized through the analysis of this three-dimensional point cloud, so as to realize the interaction. Including lidar swing module, 3D point cloud reconstruction module, graphic module, gesture recognition and feature extraction module, multi-frame comparison module and interactive control module. The data is obtained by the swinging lidar, and the period information is obtained in the data of the 3D point cloud reconstruction module, so as to estimate the third dimensional data and obtain the 3D point cloud. Graph the point cloud of each cycle through the graphing module. In the gesture recognition and feature extraction module, the trained network is used for gesture recognition, and feature extraction is performed in the graphed graph. The multi-frame comparison module recognizes gestures with long and multi-gesture combinations. The interactive control module completes intention identification and interactive control. The invention has the advantages of simple structure, high precision and high cost performance. The interaction method provided by the present invention can be applied to various occasions and applications, and has high flexibility.

The present invention specifically adopts the following technical solutions:

An augmented dimensional gesture recognition and interaction system based on two-dimensional laser radar, characterized in that it includes: a laser radar swing module, a three-dimensional point cloud reconstruction module, a graphical module, a gesture recognition and feature extraction module, a multi-frame comparison module and an interaction module control module;

The lidar swing module controls the lidar to perform a reciprocating swing with a fixed angle on a swing plane that has an included angle with the rotation plane;

The three-dimensional point cloud reconstruction module takes the scanning data of the laser radar as input, and performs three-dimensional point cloud reconstruction, including: a period calculation module, a third-dimensional interpolation module, a coordinate conversion module, and a region identification module; the period calculation module is used for calibration and Record the start of each swing cycle of the lidar; the third-dimensional interpolation module is used to estimate the swing angle of each two-dimensional point to obtain the third-dimensional component; the coordinate conversion module is used to convert polar coordinates into right angles Coordinates; the area recognition module is used to distinguish points located in the gesture recognition area among all scanning points for gesture recognition;

The graphic module uses the information carried by the three-dimensional point cloud to graphic the point cloud that has been subjected to third-dimensional interpolation and has been selected by the area recognition module, and retains the information of the gesture, including the position and shape of the gesture, to ensure that the subsequent gesture recognition and The feature extraction module can obtain accurate information.

The gesture recognition and feature extraction module recognizes the gesture through a neural network trained by machine learning through a graphical point cloud, and extracts the position of the gesture and the angle feature of the gesture;

The multi-frame comparison module compares continuous actions or actions composed of multiple gestures according to the gestures recognized by the gesture recognition and feature extraction module;

The interaction control module converts the results of the gesture recognition and feature extraction module and/or the multi-frame comparison module into corresponding intentions, and completes the interaction control according to a preset interaction mode. It can be combined with other supporting software and hardware to complete the expected interaction.

Preferably, the rotation plane and the swing plane are perpendicular to each other.

Preferably, the third-dimensional interpolation module inserts the value θ of the swing angle, and the formula is as follows:

Among them, N is the total number of points scanned in one cycle, n is the number of scan points of the current scan point in the entire cycle, and θ _max is the maximum swing angle.

Preferably, the gesture recognition area is a rectangular parallelepiped area. It can be mapped to a screen or other field of view and interact with the gesture position and screen through gestures.

Preferably, the graphic module divides the cuboid recognition area into several levels from the bottom height to the top height, corresponding to several grayscale values, and determines the height information of the point cloud according to its ratio from the bottom height to the top height. It is one of several grayscale values, and a pixel of the grayscale value converted by z is extended at the position of x and y to form a grayscale image.

Preferably, the interactive control module defines a confirmation operation surface in the rectangular area. That is, in some operations, the operation is only confirmed when the hand exceeds this surface; the control module sets an interactive language library according to the content to be controlled, and the interactive language is an interaction method recognized and used by both the interactive control module and the object to be interacted with. And the corresponding function can be realized in this way. If the input of the gesture exists in the interactive language library, the corresponding interactive control is implemented through the interactive control module.

Preferably, the lidar swing module only provides swing, but does not provide the data of the swing angle. The device includes: a lidar fixing device, a support, a movable sleeve and a turntable; the lidar fixing device is mounted on the first hinge through a first hinge. The rotating plane of the turntable is perpendicular to the support; the sleeve end of the movable sleeve is eccentrically connected to the turntable, and the connecting rod end is connected to the edge of the lidar fixing device through a second hinge.

Through a specially designed structure, a certain period of the swing cycle can cause a sudden change in the distance of a part of the scanned points. The period calculation submodule is made to scan and obtain the period by analyzing the periodic sudden change in the distance of a part of the scanned points.

Preferably, the period calculation module calibrates the start of each swing period of the lidar based on the first opening of the lidar fixing device in the scanning direction of the lidar and the second opening at the corresponding position on the support.

Preferably, in the working process of the lidar swing module and lidar, the working process of the 3D point cloud reconstruction module includes the following steps:

Step S1: The cycle calculation module analyzes the collected scan points, and takes the maximum jump scan point generated when the positions of the first opening and the second opening coincide as a reference, and takes the adjacent two maximum jump points as a reference. As a cycle, record the total number of points scanned in a cycle;

Step S2: The three-dimensional interpolation module uses the following formula to perform interpolation:

Among them, θ is the value of the swing angle, N is the total number of points scanned in a cycle, n is the scan point of the current scanning point in the entire cycle, and θ _max is the maximum swing angle;

Step 3: The coordinate conversion module is based on the polar coordinates formed by the lidar swing angle θ, the lidar rotation angle α, the lidar ranging distance l, and the coordinate conversion formula:

x=lcosαsinθ,y=lsinα,z=lcosαcosθ;

Convert polar coordinate system to Cartesian coordinate system;

Step 4: The area identification module identifies points located in the gesture identification area.

Compared with the prior art, the present invention and its preferred scheme have the following beneficial effects:

1. The construction of the solution of the present invention is based on two-dimensional laser radar as a measurement sensor, which is relatively cheap, has high data accuracy, and is reliable in measured data;

2. The present invention does not require the swing module to provide the third dimension, namely the swing angle, and completely relies on independent two-dimensional laser radar data for three-dimensional scanning. The swing accessory can be a simple mechanical structure, and no additional data measurement and transmission are required. Parts, make the system hardware system more simplified and reduce the cost;

3. The present invention can not only perform gesture recognition, but also obtain gesture position and angle information, so that the interaction can be more refined;

4. The interaction method described in the present invention can be applied to various occasions and applications, and has high flexibility.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments:

Fig. 1 is a system structure block diagram of an embodiment of the present invention;

2 is a schematic diagram of a lidar swing module according to an embodiment of the present invention;

3 is a schematic diagram of an angle representation and a rotation direction according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an interaction area for gesture recognition according to an embodiment of the present invention;

In the figure: 1- The first hinge; 2- The second hinge; 3- The sliding sleeve; 4- The turntable; ; 8- The opening on the lidar fixing device (the first opening); 9- lidar rotation mode; 10- lidar swing mode; 11- coordinate origin; 12- confirm the operation surface; 13- identify the area.

Detailed ways

In order to make the features and advantages of this patent more obvious and easy to understand, the following specific examples are given and described in detail as follows:

As shown in FIG. 1 , the system for the 2D lidar-based dimensional augmented gesture recognition and interaction method provided in this embodiment includes: a lidar swing module, a 3D point cloud reconstruction module, a graphing module, and a gesture recognition and feature extraction module , a multi-frame comparison module and an interactive control module, where:

The lidar swing module is shown in Figure 2. The lidar is attached to the lidar fixing device 7 with a light-transmitting area. The swing angle is zero degrees, and the lidar swings around the first hinge 1 . The laser rotation method is shown in 9, and the laser wobble method is shown in 10. The swing axis of the rotating radar swing structure and the lidar rotation axis intersect vertically, that is, the rotation axis of the first hinge 1 passes through the lidar rotation axis to ensure that the coordinate calculation is simplified. The second hinge 2 acquires a component of the circular motion in one direction and transmits it to the lidar fixing device 7 and then to the lidar 5 . The sliding sleeve 3 overcomes the distance change between the connection points while transmitting the rotation. The turntable 4 rotates at a constant speed and provides it with a motion component of the circular motion.

As a preferred way, the laser radar 5 of the solution in this embodiment adopts the Silan RPLIDAR A2, and uses the XH2.54-5P standard male plug to connect the lidar, and then connects to the computer with a USB data cable through the module matching the Silan RPLIDAR A2. connected, and the scanned point information enters the 3D point cloud reconstruction module.

The three-dimensional point cloud reconstruction module includes: a period calculation module, a third-dimensional interpolation module, a coordinate conversion module, and a region identification module. The scan data from the data line is first received and stored, and then the 3D point cloud is reconstructed. The specific steps are:

Step 1: The period calculation module works, analyzes and stores the scanning points, because of the special structure of the lidar fixture 7, when the lidar rotates to the angle of the position of the opening 8 of the lidar fixture, there will be a continuous value. Jump to a higher value, when the opening 8 on the fixture 7 coincides with the opening 6 used to calculate the cycle, the jump is the largest, the cycle is obtained from the two maximum jump points, and the scan in one cycle is recorded The total number of points reached.

Step 2: The third-dimensional interpolation module works, and the following formula is used to perform interpolation.

其中θ的意义如图3所示。

The meaning of θ is shown in Figure 3.

Among them, N is the total number of points scanned in one cycle, n is the number of scan points of the current scan point in the entire cycle, and θ _max is the maximum swing angle whose meaning is the number of angles shown by mark 10 in Figure 2.

Step 3: The coordinate conversion module works. As shown in Figure 3, the lidar swing angle θ, the lidar rotation angle α, and the lidar ranging distance are l. In FIG. 2 , the lidar swing angle θ corresponds to the mark 10 , and the lidar rotation angle α corresponds to the mark 9 . θ, α, l constitute polar coordinates. The specific coordinate conversion method is:

x=lcosαsinθ, y=lsinα, z=lcosαcosθ. Convert the polar coordinate system to the rectangular coordinate system by the above formula.

Step 4: The area identification module works. As shown in Figure 4, the coordinate origin 11 is the position of the lidar, and its detectable area is the shape of a sector rotated 360 degrees around the vertex, but it is only recognized in the identification area 13. Confirmation is performed only when the gesture position exceeds the confirmation operation surface 12 .

The graphic module converts the height z of the scanned point after coordinate conversion into 256 grayscale values, and extends a pixel of the grayscale value converted from z at the position of x and y. The final image is The similar form of the length h1 and the width h2 in Fig. 4 is expanded, and the gray value of the pixel point is converted from the height z. The formula for gray value conversion is as follows:

其中G表示灰度值，其大小为0到1；h3如图4所示为长方体形识别区域的高度，z'为长方体形识别区域的底部高度。

Among them, G represents the gray value, and its size ranges from 0 to 1; h3 is the height of the cuboid-shaped identification area as shown in Figure 4, and z' is the bottom height of the cuboid-shaped identification area.

The Gesture Recognition and Feature Extraction module analyzes a cycle of point clouds that have been graphed by the graphing module. A gesture recognition network with high accuracy is trained by means of machine learning using a graphical point cloud of what gesture has been marked. In the solution provided in this embodiment, no new machine learning algorithm is proposed, and the existing Some machine learning algorithms related to image recognition can achieve the purpose of the present invention. The gesture recognition network does not directly give what the gesture is, but gives the possible probability of each gesture. If the probability of all gestures does not exceed a set value, it will be recognized as other gestures and not processed.

As a preferred manner, the marked gestures may include relatively static fists, open palms, closed palms, single-finger pointing, and sliding and poking relative motions.

As a preferred method, the information mainly extracted by feature extraction is the gesture position, the color of the grayscale image contains the up and down information of the gesture, and the position of the gesture in the grayscale image contains the left and right and front and rear information of the gesture. The change of grayscale and the position of the gesture also contain the angle information of the gesture, which can also be extracted if necessary.

As a preferred way, the labeling of gestures can be done manually to improve the accuracy of labeling.

The multi-frame comparison module adopts the method of storage comparison. First, it is set that some operations are composed of several gestures. If the multi-frame comparison module finds such a combination, it will send the information to the interactive control module. Mainly suitable for relative motion gestures such as swipe and poke.

The interactive control module receives the gesture recognition and feature extraction module and the multi-frame comparison module. If you want to display gestures, you can also use the output and input of the graphics module to display gestures. The main function lies in intent recognition and interactive control.

As a preferred way, the interactive control module needs to set an interactive language library according to the content to be controlled. The interactive language is an interactive mode recognized and used by both the interactive control module and the object to be interacted with, and corresponding functions can be realized in this way. . If the input of the gesture exists in the interactive language library, the corresponding interactive control is implemented through the interactive control module.

As a preferred way, a confirmation operation surface is set in the cuboid shape recognition area, as shown in the mark 12 in FIG. 4 , that is, the operation is confirmed only after the hand exceeds this surface, especially for the operation of clicking, first of all After confirming the operation surface, adjust the position, and then confirm the operation beyond the confirmation operation surface. The interaction control module needs to confirm the operation through information such as the position of the gesture.

This patent is not limited to the above-mentioned best embodiment, anyone can come up with other various forms of two-dimensional lidar-based augmented gesture recognition and interaction systems and methods under the inspiration of this patent. Anyone who applies for a patent according to the present invention Equivalent changes and modifications made in the scope shall fall within the scope of this patent.

Claims (9)

1. a dimensional increase gesture recognition and interaction system based on two-dimensional laser radar, is characterized in that, comprises: laser radar swing module, three-dimensional point cloud reconstruction module, graphic module, gesture recognition and feature extraction module, multi-frame comparison module and interactive control module; The lidar swing module controls the lidar to swing at a fixed angle on a swing plane that has an included angle with the rotation plane; The three-dimensional point cloud reconstruction module takes the scanning data of the laser radar as input, and performs three-dimensional point cloud reconstruction, including: a period calculation module, a third-dimensional interpolation module, a coordinate conversion module, and an area identification module; the period calculation module is used for calibration and Record the start of each swing cycle of the lidar; the third-dimensional interpolation module is used to estimate the swing angle of each two-dimensional point to obtain the third-dimensional component; the coordinate conversion module is used to convert polar coordinates into right angles Coordinates; the area recognition module is used to distinguish points located in the gesture recognition area among all scanning points; The graphic module uses the information carried by the three-dimensional point cloud to graphic the point cloud that has been subjected to third-dimensional interpolation and selected by the area identification module, and retains the information of the gesture, including the position and shape of the gesture; The gesture recognition and feature extraction module recognizes the gesture through a neural network trained by machine learning through a graphical point cloud, and extracts the position of the gesture and the angle feature of the gesture; The multi-frame comparison module compares continuous actions or actions composed of multiple gestures according to the gestures recognized by the gesture recognition and feature extraction module; The interaction control module converts the results of the gesture recognition and feature extraction module and/or the multi-frame comparison module into corresponding intentions, and completes the interaction control according to a preset interaction mode. 2 . The two-dimensional laser radar-based augmented dimensional gesture recognition and interaction system according to claim 1 , wherein the rotation plane and the swing plane are perpendicular to each other. 3 . 3. The dimensional increase gesture recognition and interaction system based on two-dimensional laser radar according to claim 1, is characterized in that: The third-dimensional interpolation module inserts the value θ of the swing angle, and the formula is as follows:

Among them, N is the total number of points scanned in one cycle, n is the number of scan points of the current scan point in the entire cycle, and θ _max is the maximum swing angle. 4 . The two-dimensional laser radar-based augmented dimensional gesture recognition and interaction system according to claim 1 , wherein the gesture recognition area is a rectangular parallelepiped area. 5 . 5. The dimensional augmented gesture recognition and interaction system based on two-dimensional laser radar according to claim 4, wherein the graphic module divides the cuboid recognition area into several levels from the bottom height to the top height, corresponding to several Gray value, the height information of the point of the point cloud is determined as one of several gray values according to its ratio from the bottom height to the top height, and a z-converted one is extended at the x and y positions. Pixels of gray value, forming a grayscale image. 6 . The two-dimensional lidar-based augmented dimensional gesture recognition and interaction system according to claim 4 , wherein the interaction control module defines a confirmation operation surface in the cuboid region. 7 . 7. The two-dimensional laser radar-based augmented dimensional gesture recognition and interaction system according to claim 2, wherein the laser radar swing module comprises: a laser radar fixing device, a support, a movable sleeve and a turntable; The lidar fixing device is installed on the support through a first hinge; the rotation plane of the turntable is perpendicular to the support; the sleeve end of the movable sleeve is eccentrically connected to the turntable, and the connecting rod end is connected to the lidar through a second hinge The edge of the fixture. 8. The augmented dimensional gesture recognition and interaction system based on two-dimensional lidar according to claim 7, characterized in that: the cycle calculation module calibrates the start of each swing period of the lidar, based on the lidar fixing device in the laser A first opening in the radar scanning direction, and a second opening in a corresponding position on the support. 9 . The recognition and interaction method of a two-dimensional laser radar-based dimensionally augmented gesture recognition and interaction system according to claim 8 , wherein, in the working process of the laser radar swing module and the laser radar, the three-dimensional point cloud The working process of the rebuild module includes the following steps: Step S1: The cycle calculation module analyzes the collected scan points, and takes the maximum jump scan point generated when the positions of the first opening and the second opening coincide as a reference, and takes the adjacent two maximum jump points as a reference. As a cycle, record the total number of points scanned in a cycle; Step S2: The three-dimensional interpolation module uses the following formula to perform interpolation:

Among them, θ is the value of the swing angle, N is the total number of points scanned in a cycle, n is the scan point of the current scanning point in the entire cycle, and θ _max is the maximum swing angle; Step 3: The coordinate conversion module is based on the polar coordinates formed by the lidar swing angle θ, the lidar rotation angle α, the lidar ranging distance l, and the coordinate conversion formula: x=lcosαsinθ,y=lsinα,z=lcosαcosθ; Convert polar coordinate system to Cartesian coordinate system; Step 4: The area identification module identifies points located in the gesture identification area.

Images