KR20180074124A - Method of controlling electronic device with face recognition and electronic device using the same - Google Patents

Abstract

The present invention provides a method for controlling an electronic device through face recognition by determining expression, movement or a direction of a user face and an electronic device performing the same. The method for controlling an electronic device through face recognition comprises the following steps of: capturing an image in which the user face is included; recognizing the user face from the captured image; determining at least one among the expression, the movement or the direction of the recognized face and generating face information; and performing a control command corresponding to the face information generated based on a pre-stored mapping table.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method of controlling an electronic device using facial recognition,

The present invention relates to a method of controlling an electronic device through facial recognition and an electronic apparatus for performing the method. More particularly, the present invention relates to a method of determining an expression, movement or direction of a user face recognized in a captured image, To a method of controlling an electronic device corresponding to a facial expression, a movement, or a direction, and an electronic apparatus for performing the method.

As the face recognition technology develops, the face recognition technology performs various functions such as the auto focus function of the camera, the security function of the door, and the like.

Although face recognition technology is likely to induce users' interest, the use of face recognition technology as entertainment has not been actively pursued.

Accordingly, there is a need for a variety of entertainment services using a technique of controlling a screen of an electronic device using a face recognition technology.

In addition, if the user can control the electronic device by recognizing the user's face without attaching a sensor such as a gyro sensor, an acceleration sensor, or the like to the user, the user can move the electronic device It will be possible to control it.

It is an object of the present invention to provide a method of controlling an electronic device by judging a facial expression, a movement or a direction of a user's face and an electronic apparatus for performing the method.

According to another aspect of the present invention, there is provided a method of controlling an electronic device through face recognition, the method comprising: capturing an image including a user's face; Recognizing a user's face on the photographed image; Generating face information by determining at least one of a facial expression, a movement, and a direction of the recognized face; And performing a control command corresponding to the generated face information based on the pre-stored mapping table.

The step of performing the control command may include: determining whether the generated face information is present in a pre-stored mapping table; Selecting a control command corresponding to the generated face information in the pre-stored mapping table when the face information is present in the pre-stored mapping table; And performing the selected control command.

In addition, the pre-stored mapping table may be a table in which face information composed of information on facial expressions, motions or directions, and control commands for controlling electronic devices are mapped on a one-to-one basis.

The step of determining at least one of a facial expression, a movement, and a direction of the recognized face may include: detecting a plurality of feature points from the recognized face; And determining at least one of a facial expression, a movement, and a direction by analyzing a correlation between the plurality of detected minutiae points.

According to another aspect of the present invention, there is provided an electronic device including: a camera that captures an image including a face of a user; And a controller for recognizing a face of the user in the photographed image and generating face information by judging at least one of a facial expression, a motion, and a direction of the recognized face, and based on the pre-stored mapping table, And a control unit for executing the command.

As described above, according to various embodiments of the present invention, the user is able to perform unrestrained control of the electronic device through facial expressions, movements or directions.

1 is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present invention;
2 is a diagram for describing an embodiment for detecting facial feature points, according to an embodiment of the present invention;
3 and 4 are diagrams for explaining a method for controlling an electronic device through face recognition according to various embodiments of the present invention,
5 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

The terms used in the embodiments of the present disclosure will be briefly described, and the embodiments will be described in detail.

The terms used in the embodiments of the present disclosure have selected the currently widely used generic terms possible in light of the functions in this disclosure, but these may vary depending on the intentions or precedents of those skilled in the art, the emergence of new technologies, and the like . Also, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, the meaning thereof will be described in detail in the description of the corresponding embodiments. Therefore, the terms used in the embodiments should be defined based on the meaning of the term, not on the name of a simple term, and on the contents of the embodiments throughout.

In the embodiments of the present disclosure, terms including ordinal numbers such as first, second, and so on can be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Moreover, in the embodiments of the present disclosure, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

Also, in the embodiments of the present disclosure, terms such as "comprise" or "have ", etc. are intended to specify that there is a feature, number, step, operation, component, Steps, operations, elements, components, or combinations of elements, numbers, steps, operations, components, parts, or combinations thereof.

Further, in the embodiments of the present disclosure, "module" or "module" performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. Also, a plurality of "modules" or a plurality of "parts" may be embodied as at least one processor integrated into at least one module, except "modules" or "modules "

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, the present disclosure will be described with reference to the drawings. 1 is a diagram showing a configuration of an electronic device according to an embodiment of the present invention. 1, the electronic device 100 includes a camera 110, a display unit 120, and a control unit 130. [ Meanwhile, the electronic device 100 according to an embodiment of the present invention may be implemented by various electronic devices such as a desktop PC, a notebook PC, a tablet PC, a kiosk, a smart TV, a smart phone, an electric wheelchair, and the like.

The camera 110 is configured to shoot an image. In particular, the camera 110 may capture an image including the face of at least one user.

The display unit 120 displays at least one of a video frame processed by an image processing unit (not shown) and various screens generated by a graphics processing unit (not shown), the image data received from an image receiving unit (not shown). The control unit 130 controls the overall operation of the electronic device 100 using various programs stored in a storage unit (not shown). The controller 150 may include a RAM, a ROM, a graphics processor, a main CPU, first to n-th interfaces, a bus, and the like. The RAM, the ROM, the graphics processor, the main CPU, Lt; / RTI >

Particularly, when the image including the user's face is photographed through the camera 110, the control unit 130 recognizes the user's face in the photographed image and determines at least one of the facial expression, movement, or direction of the recognized face Face information can be generated. The control unit 130 may execute a control command corresponding to the face information generated based on the pre-stored mapping table. At this time, the control command may be a hardware control of the electronic device as well as a control of the displayed UI.

Specifically, the controller 130 can recognize the face of the user by distinguishing the face and the background of the user based on the brightness, motion, color, and eye position estimation of the photographed image. For example, the control unit 130 may generate a separate learning result for data specifying a face region (generally, a rectangular region) for a person included in a plurality of images having different brightness, face shape, age, The user's face can be recognized through the user's face. As the number of image data for learning increases, the accuracy of the learning result increases. As a result, the user's face can be detected more accurately in the captured image.

When the face of the user is recognized, the controller 130 displays the face shape 210, the eyebrow 220, the eyes 230, the nose 240, and the mouth 250) of the minutiae can be generated. These feature points can be detected by using separate learning results for data indicating the positions of facial features, eyebrows, eyes, nose, and mouth with respect to multiple faces, and as the data for learning increases, It is possible to increase the accuracy of detecting feature points in the face.

The controller 130 may determine at least one of facial expression and facial motion by analyzing the correlation between the detected plurality of feature points. For example, the controller 130 determines whether or not the eye is closed by analyzing the positional correlation between the minutiae allocated to the eyes, analyzes the positional relationship between the minutiae allocated to the mouth, and determines whether the mouth is opened And determine at least one of facial expression or facial motion based on the shape of the eye, eyebrow, mouth, and the like.

Then, the controller 130 can determine the direction of the face based on the ratio of eyes, eyebrows, nose, mouth, and the like. For example, when the length between the nose and the left eye is longer than the length between the nose and the right eye, the controller 130 can determine that the direction of the user's face is directed to the right, and the length between the nose and the left eye is The higher the ratio of the length between the nose and the right eye, the larger the angle can be judged to be directed to the right.

Then, the controller 130 can generate face information based on the facial expression, motion, or direction of the determined face.

The control unit 130 may execute a control command corresponding to the face information generated based on the pre-stored mapping table.

Specifically, the controller 130 can determine whether the generated face information is present in the pre-stored mapping table. If the generated face information is present in the pre-stored mapping table, the controller 130 may select a control command corresponding to the face information from the pre-stored mapping table and execute the selected control command.

3, if the face information generated through face recognition is information whose face is oriented in the upper left direction, the controller 130 determines whether the face direction is the upper left direction It is possible to determine whether or not the information directed to the pre-stored mapping table exists. The control unit 130 displays an icon 310 displayed on the display unit 310. When the icon 310 is displayed in a one-to-one correspondence with the icon 310, And moves the icon 310 in the upper left direction to control the display unit 120 to display the control command. If the blinking motion information is stored in the pre-stored mapping table and the control command for selecting the area where the icon is located is stored in a one-to-one correspondence with each other, the control unit 130 displays a control command for selecting the area where the icon 310 is located And select a UI element 330 in which the icon 310 is located. On the other hand, one-to-one correspondence is made between face information in which the direction of the user's face faces the first direction and a control command to move the icon in the first direction, and face-blinking face information and control commands for selecting UI elements correspond one- Stored in the table corresponds to one embodiment, but is not limited thereto.

According to another embodiment of the present invention, as shown in FIG. 4, the game can be executed by controlling the movement of the character through face recognition. For example, the pre-stored mapping table may include facial information for opening the mouth, a control command for the character 410 to jump, facial information for blinking eyes, a control command for the character 410 to move forward, facial information for raising and lowering eyebrows, 410 may be a one-to-one correspondence mapping table. When an image including a face is photographed through the camera 120, the control unit 130 recognizes the face in real time, and determines facial expression, movement, or direction of the face to generate face information in real time. If the generated face information is the face information of the mouth, the control unit 130 can control the character 410 to jump by selecting a control command to jump the character 410. If the generated face information is eye-blinking face information, the controller 130 may control the character 410 to move forward by selecting the control command for the character 410 to move forward. If the generated face information is facial information for raising and lowering the eyebrows, the control unit 410 can control the character 410 to run backward. When the character 410 faces the obstacle 420, which is running forward, the user can jump over the obstacle and play forward by controlling the face recognition.

According to still another embodiment of the present invention, the pre-stored mapping table includes a first control information for moving the first wheelchair information and the first wheelchair information, a control command for moving the wheelchair forward, a control command for turning the second wheelchair rightward, A control command to make a left turn and a control command to stop the third wheelchair and the third face information may be a one-to-one mapping table. In this case, the user can control the electric wheelchair using facial expressions, motions or directions.

Hereinafter, a method of controlling the electronic device 100 that can be controlled through face recognition according to an embodiment of the present invention will be described with reference to FIG.

First, the electronic device 100 captures an image including a user's face (S510). The electronic device 100 may photograph not only an image including one user's face but also an image including multiple user's faces.

Then, the electronic device 100 recognizes the user's face on the photographed image (S520). Specifically, the electronic device 100 can recognize the user's face in the captured image based on the previously stored learning data.

Then, the electronic device 100 determines at least one of a facial expression, a movement, and a direction of the recognized face to generate facial information (S530). Specifically, the electronic device 100 can generate face information by detecting at least one of facial expression, movement, or direction of a recognized face by detecting a plurality of feature points from the recognized face and analyzing a correlation between the detected feature points have. For example, the electronic device 100 may determine facial expression of the recognized facial expression as a blank facial expression, blinking of the eyes, or facial information to the right, thereby generating facial information.

Then, the electronic device 100 performs a control command corresponding to the face information generated based on the pre-stored mapping table (S540). Specifically, the electronic device 100 determines whether or not the generated face information is present in the pre-stored mapping table. If the generated face information is present in the pre-stored mapping table, It is possible to execute the corresponding control command.

According to various embodiments of the present invention as described above, the user can control the electronic device by taking at least one of facial expressions, motions or directions in a non-restrained state.

Meanwhile, the above-described method can be implemented in a general-purpose digital computer that can be created as a program that can be executed by a computer and operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, disclosure methods should be considered from an illustrative point of view, not from a restrictive point of view. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Electronic device
110: camera
120:
130:

Claims (5)

A method for controlling an electronic device through face recognition,
A step of photographing an image including a user's face;
Recognizing a user's face on the photographed image;
Generating face information by determining at least one of a facial expression, a movement, and a direction of the recognized face;
And performing a control command corresponding to the generated face information based on the pre-stored mapping table. The method according to claim 1,
Wherein performing the control command comprises:
Determining whether the generated face information is present in a pre-stored mapping table;
Selecting a control command corresponding to the generated face information in the pre-stored mapping table when the face information is present in the pre-stored mapping table; And
And performing the selected control command. The method of controlling an electronic device according to claim 1, 3. The method of claim 2,
The pre-
Wherein face information composed of information on facial expressions, motions or directions and control commands for controlling electronic devices are mapped on a one-to-one basis. The method according to claim 1,
Determining at least one of a facial expression, a movement, and a direction of the recognized face,
Detecting a plurality of feature points from the recognized face; And
And determining at least one of a facial expression, a motion, and a direction by analyzing a correlation between the detected plurality of feature points. A camera for photographing a video image including a user's face;
And a controller for recognizing a face of the user in the photographed image and generating face information by judging at least one of a facial expression, a motion, and a direction of the recognized face, and based on the pre-stored mapping table, And a control unit for executing the command.

Images