KR20230023520A - Long-distance motion gesture recognition apparatus - Google Patents

Abstract

The present invention relates to a long-distance motion gesture recognizing device comprising: a camera which obtains an image for a user; an image processing unit which obtains data for extracting a gesture of the user by image processing for the image obtained from the camera; a command operation extracting unit which extracts a corresponding command by performing voice recognition based on deep learning from the data obtained by the image processing unit; a controlling unit which outputs a controlling signal corresponding to the command extracted by the command operation extracting unit; and a transmitting unit which transmits the controlling signal outputted from the controlling unit to a controlled device in a wired or wireless method. The present invention increases a recognizing rate; greatly contributes to improvement of a gesture recognizing rate from a long distance; and is applied to smart home appliances and smart home electronic products which are widely used in conjunction with a smartphone application or smart home systems such as smart things, a home assistant, a home kit, etc. to increase use convenience.

Description

Long-distance motion gesture recognition apparatus {Long-distance motion gesture recognition apparatus}

The present invention relates to a long-distance motion gesture recognition device, and more particularly, to a long-distance motion gesture recognition device that can greatly contribute to improving the gesture recognition rate at a long distance as well as to increase the command recognition rate for controlling a controlling device. will be.

In general, the development and distribution of products and services that combine voice recognition technology with artificial intelligence technology, a key element of the 4th industrial revolution, are rapidly expanding. Among them, artificial intelligence speakers are attracting attention as a hub of home IoT because they can remotely control other home appliances in the IoT era by simply giving voice commands.

As a technology related to conventional artificial intelligence speakers, the "artificial intelligence speaker system" of Korean Patent Registration No. 10-2266320 includes an internal microphone mounted on a main body and voice instructions among sound signals input through the internal microphone. a main device having a main terminal control unit that outputs response information to the corresponding voice instruction content through an internal speaker, and a main terminal communication unit that is controlled by the main terminal control unit and communicates with registered user terminals; It is separated from the main body and when an external noise blocking mode execution instruction signal is received from the main device through the sub-terminal communication unit, an external canceling interference signal capable of canceling interference with the external acoustic signal received from the external microphone is generated. and a sub device that generates and outputs the output through an external speaker, wherein the main device supports settings of the external noise blocking mode, sleep mode, parenting mode, and study mode, and when a signal for setting the sleep mode to the sleep mode is received. The sound information received through the internal microphone is stored and recorded in the storage unit, and when a setting instruction signal for the parenting mode is received, when a crying sound is detected in the sound information received through the internal microphone, the registered user terminal A message informing of the occurrence of a crying sound is transmitted, and when a signal indicating the setting of the study mode is received, an internal destructive interference signal is generated to remove the sound signal received through the internal microphone by destructive interference so as to operate the internal speaker. output can be processed.

However, such a prior art not only has a low voice recognition rate and the hassle of producing a voice, but also has problems in that it is difficult to use for people with disabilities who cannot speak.

In order to solve the problems of the prior art as described above, the present invention not only increases the command recognition rate for controlling the controlling device, but also greatly contributes to the improvement of the gesture recognition rate at a long distance, and the smartphone app or smartthings, Its purpose is to increase the convenience of use by allowing it to be applied to smart home appliances or smart home electronic products that are widely used in conjunction with smart home systems such as Home Assistant and HomeKit.

Other objects of the present invention will be easily understood through the description of the following embodiments.

In order to achieve the above object, according to one aspect of the present invention, a camera for obtaining an image of the user; an image processing unit that obtains data for extracting the user's gesture by image processing on the image obtained from the camera; a command action extraction unit for extracting a corresponding command by performing deep learning-based voice recognition from the data acquired by the image processing unit; a control unit outputting a control signal corresponding to the command extracted by the command operation extraction unit; and a transmission unit configured to transmit the control signal output from the control unit to the controlled device by wire or wirelessly.

The image processing unit obtains an image of the user's chest, including the face, as data for extracting the user's gesture, and the command action extraction unit uses the data obtained by the image processing unit to obtain the image including the face. To use the image up to the chest as an input for deep learning, face recognition is performed to secure an ROI area through the face, and specific gestures are obtained through a deep learning model by using specific frame time series data as input for LSTM and CNN. By classifying as , it is possible to extract a command indicated by a specific gesture.

During deep learning, the command action extraction unit collects images frame by frame and repeats the process in real time, and allows a predetermined number of frames to overlap during the repetition process.

a microphone provided to input a user's voice; and a voice processing unit that processes voice input to the microphone and obtains data for command extraction, wherein the command action extraction unit includes a plurality of pre-stored voices of the user from data obtained by the voice processing unit. Compared with the reference voice, the matched reference voice may extract a corresponding command.

a main user recognition unit for recognizing a gesture of a main user from the data obtained by the image processing unit, so that the command motion extraction unit extracts a corresponding command by performing deep learning-based voice recognition from the gesture of the main user; and a commanding motion extraction unit for sequentially identifying a device indicated by the hand from the main user's gesture recognized by the main user recognition unit within the image obtained by the camera, for a specified time period; The command action extraction unit extracts a corresponding command by performing deep learning-based voice recognition from an operation gesture given by the main user by hand, and the control unit extracts the command extracted by the command operation extraction unit. It can be controlled to transmit a control signal corresponding to the command to the device identified immediately before.

According to the long-distance motion gesture recognition device according to the present invention, it not only increases the command recognition rate for controlling a controlling device, but also can greatly contribute to improving the gesture recognition rate at a long distance, and can be used in smartphone apps, smart things, home assistant, It has the effect of enhancing the convenience of use by allowing it to be applied to smart home appliances or smart home electronic products that are widely used in conjunction with smart home systems such as HomeKit.

1 is a perspective view illustrating a long-distance motion gesture recognition device according to an embodiment of the present invention.
2 is a configuration diagram illustrating a long-distance motion gesture recognition apparatus according to an embodiment of the present invention.
3 is a perspective view illustrating a long-distance motion gesture recognition device according to another embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood in such a way as to include all changes, equivalents, or substitutes included in the technical spirit and scope of the present invention, and may be modified in various other forms. However, the scope of the present invention is not limited to the following examples.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, the same reference numerals will be assigned to the same or corresponding components regardless of reference numerals, and redundant descriptions thereof will be omitted.

1 is a configuration diagram showing a long-distance motion gesture recognition device according to an embodiment of the present invention, and FIG. 2 is a perspective view showing how the long-distance motion gesture recognition device according to an embodiment of the present invention is used.

1 and 2, a long-distance motion gesture recognition device 10 according to an embodiment of the present invention includes a camera 11, an image processing unit 12, a command operation extraction unit 13, a control unit 14, and It may include a transmission unit 15, and the casing 25 is made of a cylindrical shape suitable for portable use, but is not necessarily limited thereto.

The camera 11 is to acquire an image of the user, and may be installed, for example, on the front of the casing 25 so as to acquire an image in a forward direction. In the case of a video obtained by the camera 11, for example, a video, as a series of images, a photo image is transmitted at a specific number of frames per second. can be obtained in quick succession.

The image processing unit 12 acquires data for extracting a user's gesture by image processing on an image obtained from the camera 11 . The image processing unit 12 may obtain an image of the user's chest up to the face including the user's face as data for extracting the user's gesture. The image processing unit 12 converts the user's appearance into data using an image processing technique using body characteristics of the user, such as head, face, arm, hand, leg, and foot, in the image obtained by the camera 11. It can be acquired and provided to the command operation extraction unit 13. A known technique may be applied to the image processing technique by the image processing unit 12 .

The command action extraction unit 13 extracts a corresponding command by performing deep learning-based voice recognition from the data acquired by the image processing unit 12. The command motion extraction unit 13 recognizes a moving frame, that is, a motion, not a specific sign of the hand. In addition, the command action extraction unit 13 is configured to compare the data obtained by the image processing unit 12 with reference gesture data previously stored in the memory unit 21, and extract a control signal indicated by the matched reference gesture data as a command. It could be.

The command action extraction unit 13 uses the data acquired by the image processing unit 12 to perform face recognition to utilize images up to the chest, including the face, as inputs for deep learning. By securing an ROI area, using specific frame time-series data as input to LSTM and CNN, and classifying it as a specific gesture through a deep learning model, commands indicated by the specific gesture can be extracted. In the case of gesture recognition, recognition using depth vision is performed, which is advantageous for short-distance recognition, but accuracy decreases as the distance increases. On the other hand, after face recognition, the command motion extraction unit 13 utilizes a hand motion recognition device in a specific area including the face to enable long-distance gesture recognition at a distance of about 10 m. Also, the command operation extraction unit 13 may be configured so that the embedded board operates in a local environment rather than a server through model optimization.

The command motion extraction unit 13 can apply R-CNN + LSTM technique through a 2-stage dector for gesture recognition, and proceeds with Facedection to utilize images from the face to the chest as input for deep learning. By securing the ROI area through the face, it is possible to optimize and simplify the deep learning model, reduce the model size, and increase accuracy. The command action extraction unit 13 uses specific frame time series data as an input of LSTM and CNN and classifies them into specific gestures through a deep learning model. Various motions, such as a signal, clockwise rotation, arm rolling, clapping, no gesture, etc., may be applicable. The command operation extraction unit 13 is a case of an artificial intelligence model, and proceeds with model execution through a local offline environment of its own board without going through a server. As an embedded board that can be used, for example, an OS embedded board, a micro kernel It can be configured to utilize AI calculators, FPGA calculators, and the like.

During deep learning, the command action extraction unit 13 collects images in frame units and repeats them in real time. During the repetition, a predetermined number of frames may be overlapped. For example, images are collected in frame units and repeated in real time. However, in deep learning, model recognition is performed once per specific frame time series data, and the accuracy can be strengthened by continuously accepting frames in real time and continuously performing them. This is, for example, if the video is a total of frames 1 to 120, if the frames entering the deep learning model are 30 frames and the frames discarded for continuation are 10, the model is first performed with images 1 to 30, , the second one performs the model with images from 10 to 40, and the third one can perform the model with images from 20 to 50. If this is performed continuously, a total of 9 gesture results will be displayed by performing the model with images from 90 to 120 times. This has the advantage that if the first gesture and the second gesture are the same, a more accurate algorithm can be used, such as the gesture being executed, and if the gesture is the first gesture result, the recognition speed is very fast. .

The command operation extraction unit 13 is a predetermined command according to a plurality of specific gestures, and various processes including stop, operation, and setting value change may be applicable.

The control unit 14 outputs a control signal corresponding to the command extracted by the command operation extraction unit 13.

The transmission unit 15 transmits the control signal output from the control unit 14 to the control target devices 1, 2, and 3 by wire or wirelessly. The transmission unit 15 may transmit a control signal to each of the control target devices 1, 2, and 3 in a wired manner through a cable, but is not limited thereto, and the control target devices 1, 2, and 3 pre-registered. It is possible to transmit/receive signals with the control target devices 1, 2, and 3 using various wireless communication methods including ZigBee, Bluetooth, Wi-Fi, 3G, LTE, or 5G. The control target devices 1, 2, and 3 may be smart home appliances or smart home electronic products used in conjunction with a smart home system such as a smart phone app, smartthings, home assistant, or homekit.

The long-distance motion gesture recognition device 10 according to an embodiment of the present invention may further include a microphone 16 and a voice processing unit 17, and furthermore, a scanning recognition unit 18 and a command motion extraction unit 19 ) may be further included.

The microphone 16 is provided to input the user's voice, for example, it is formed to be located on the front of the casing 25, but is not necessarily limited thereto.

The voice processing unit 17 processes voice input to the microphone 16 to obtain data for command extraction. The voice processing unit 17 may use one of various voice recognition algorithms known in the art of the present invention, for example, a hidden markov model (HMM), a Gaussian mixture model (GMM), and a support vector machine (SVM). Any one of statistical pattern recognition methods or artificial neural network models such as RNN, LSTM, DNN, and CNN, or a combination thereof may be used.

The command action extraction unit 13 compares the user's voice with a plurality of pre-stored reference voices from the data acquired by the voice processing unit 17, and extracts a corresponding command from the matched reference voice.

The main user recognition unit 18 recognizes the main user's gesture from the data acquired by the image processing unit 12, so that the command motion extraction unit 13 performs deep learning-based voice recognition from the main user's gesture You can extract the corresponding command. For example, the main user recognition unit 18 considers that the main user is closest to the camera 11 among a plurality of people photographed by the camera 11, and recognizes the closest user as the main user. Or, considering that the main user performs a gesture for a command with a large motion compared to the people around him, the user who performs the largest motion is recognized as the main user, or the main user makes a gesture required for a command more than the people around him. Considering the relatively accurate performance, it is possible to recognize the main user by various methods, such as recognizing a user with clear gesture recognition as the main user, and thereby removing noise about the motions of people around the main user. and, thereby, it is possible to extract an accurate command by the main user.

The command motion extraction unit 19 processes to sequentially identify devices that are directed by the hand from the gestures of the main user recognized by the main user recognition unit 18 within the image acquired by the camera 11 for a specified period of time. can do. When it is determined that the main user's gesture recognized by the main user recognition unit 18 points to a specific device with a hand, the pointing motion extraction unit 19 determines that the pointing accuracy among the devices 1, 2, and 3 pointed by the hand is From the highest device, it is possible to easily check whether or not the device is instructed by sequentially outputting the name of the device to the outside through the speaker 22 for a predetermined time, for example, within the range of 0.5 to 2.0 sec. let it be At this time, if the name of the output of the speaker 22 is different from the device actually pointed by the user, the user may correct the gesture so that it faces the device again. ,2,3) can be identified. The name of the device may be configured to be registered in advance as a control operation of the control unit 14 by a user's manipulation of the operation unit 23 .

The command action extraction unit 13 may extract a corresponding command by performing voice recognition based on deep learning from an action gesture given by the main user by hand. In addition, the control unit 14 may control the device identified immediately before the command extracted by the command operation extraction unit 13 to transmit a control signal corresponding to the command. Therefore, by this function, the control unit 14 and the like perform communication necessary for operation control with the user. The user identifies the control target device 1, 2, 3 by pointing with his hand, and then controls the device. Able to give orders accurately.

The long-distance motion gesture recognition device 10 according to an embodiment of the present invention may further include a memory unit 21, a speaker 22, a control unit 23, and a power supply unit 24. The memory unit 21 can store various data necessary for the operation of the command operation extraction unit 13 and the control unit 14, each command of a reference gesture, and each command of reference voice data, and the command operation extraction unit 13 ), the control unit 14, etc. can provide corresponding data upon request, and in addition, various data and programs or applications necessary for operation can be stored. The speaker 22 may be provided for functions required as an audio output corresponding to a necessary signal or an artificial intelligence speaker. The control unit 23 may be composed of a plurality of buttons, switches, or a touch panel to directly input signals necessary for operation. The power supply unit 24 supplies power required for operation, and for this purpose, it may be a rechargeable battery that can be charged by a charging circuit, or may be configured to supply power applied from a dry cell or an external power source that is detachably accommodated.

3 is a perspective view illustrating a long-distance motion gesture recognition device according to another embodiment of the present invention.

Referring to FIG. 3 , a long-distance motion gesture recognition device 30 according to another embodiment of the present invention has the same components as the long-distance motion gesture recognition device 10 according to the previous embodiment, but has a casing 35 structure. It may have a difference, and in this embodiment, a casing 35 having a fixed type structure such as a hexahedron may be used to enable a fixed type use.

A camera 31, a microphone 36, a speaker 42 and a control unit 43 may be provided on the front of the casing 35.

According to the long-distance motion gesture recognition device according to the present invention, not only can the recognition rate be increased, but it can also greatly contribute to improving the gesture recognition rate at a long distance.

According to the present invention, it is possible to improve the convenience of use by enabling application to frequently used smart home appliances or smart home electronic products in conjunction with smart home systems such as smart phone apps, smart things, home assistant, and home kit.

As described above, the present invention has been described with reference to the accompanying drawings, but various modifications and variations can be made without departing from the technical spirit of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, and should be defined by not only the claims to be described later, but also those equivalent to these claims.

1,2,3: control target device 11,31: camera
12: image processing unit 13: command motion extraction unit
14: control unit 15: transmission unit
16,36: microphone 17: voice processing unit
18: injection recognition unit 19: instructional motion extraction unit
21: memory unit 22,42: speaker
23,43: control unit 24: power supply unit
25,45: Casing

Claims (5)

a camera for obtaining an image of a user;
an image processing unit that obtains data for extracting the user's gesture by image processing on the image obtained from the camera;
a command action extraction unit for extracting a corresponding command by performing deep learning-based voice recognition from the data acquired by the image processing unit;
a control unit outputting a control signal corresponding to the command extracted by the command operation extraction unit; and
a transmission unit for transmitting the control signal output from the control unit to a control target device in a wired or wireless manner;
Including, long-distance motion gesture recognition device. The method of claim 1,
The image processing unit,
Obtaining an image of the user's chest up to the user's face as data for extracting the user's gesture;
The command operation extraction unit,
Using the data acquired by the image processing unit, face recognition is performed to secure the ROI area through the face in order to utilize the image including the face and the chest as an input for deep learning, and a specific frame time series data is obtained. A long-distance motion gesture recognition device that uses LSTM and CNN as inputs and classifies them as specific gestures through a deep learning model to extract commands indicated by specific gestures. The method of claim 1,
The command operation extraction unit,
In deep learning, a long-distance motion gesture recognition device that collects images in frame units and repeats them in real time, and overlaps a predetermined number of frames during the repetition. The method of claim 1,
a microphone provided to input a user's voice; and
A voice processing unit that processes voice input to the microphone and obtains data for command extraction;
The command operation extraction unit,
The long-distance motion gesture recognition apparatus for comparing the user's voice with a plurality of pre-stored reference voices from the data obtained by the voice processing unit, and extracting a corresponding command from the matched reference voice. The method according to any one of claims 1 to 4,
a main user recognition unit for recognizing a gesture of a main user from the data obtained by the image processing unit, so that the command motion extraction unit extracts a corresponding command by performing deep learning-based voice recognition from the gesture of the main user; and
In the image obtained by the camera, a command motion extractor for processing to sequentially identify a device that is instructed by the hand from the gesture of the main user recognized by the main user recognition unit for a specified period of time;
The command operation extraction unit,
Deep learning-based voice recognition is performed from the operation gestures after the main user gives a hand instruction to extract a corresponding command,
The control unit,
The long-distance motion gesture recognition device controls to transmit a control signal corresponding to the command to a device identified immediately before the command extracted by the command action extraction unit.

Images