JP6209067B2 - Image recognition apparatus and image recognition method - Google Patents

Description

  The present invention relates to an image recognition apparatus and an image recognition method.

  In the past, automatic sign language translation systems have focused on improving the accuracy of hand motion recognition, etc., from the perspective of how to improve input accuracy rather than the idea of simply inputting characters by predictive conversion. System research was conducted.

  For example, a user's image is photographed with two cameras, a region having a relatively large difference in pixel values of the respective images is defined as a user region, a top portion in the user region is defined as a head region, and a region sharper than the top portion is defined as a tip. A technique is known that identifies a region and determines a gesture from the movement of the hand (Patent Document 1).

JP 2009-211153 A

  By the way, the same or similar gesture operation may have a plurality of meanings. In this case, even if the accuracy of motion recognition is high, there is a problem that the meaning cannot be accurately determined.

  The present invention has been made to solve such problems, and an object thereof is to provide an image recognition apparatus and an image recognition method that can accurately determine the meaning of a gesture operation.

In order to solve the above-described problems, an image recognition apparatus according to the present invention recognizes a shooting target gesture based on an image acquisition unit that acquires a shooting target as image data and the image data acquired by the image acquisition unit. Based on the first recognition means and the image data acquired by the image acquisition means, the movement of the mouth of the shooting target in the gesture recognized by the first recognition means is recognized, and the movement of the mouth of the shooting target is recognized, Second recognition means for recognizing the number of sounds based on the movement of the mouth in the gesture, semantic candidates corresponding to the recognition result of the gesture by the first recognition means, and recognition of the movement of the mouth to be imaged by the second recognition means based on the results, and determining means for determining the meaning of the gesture, and output means for outputting the meaning of the gesture the determination means determines, the number of sounds that have been recognized by the second recognizing means Unused information holding means for holding, as unused information, difference information between the number of sounds recognized by the second recognizing means and the number of meaning characters judged by the judging means when the number of meaning characters judged by the judging means is exceeded When there is unused information in the gesture immediately before the gesture to be recognized, the determination unit adds the unused information to the recognized number of sounds recognized by the second recognition unit, and the meaning of the gesture To judge,

In the image recognition method according to the present invention, in the image recognition method of recognizing an image recognition apparatus gestures imaging target, an image acquisition step of acquiring a photographed subject image data, image data acquired by the image acquisition step Based on the first recognition step for recognizing the gesture to be photographed, and based on the image data acquired by the image acquisition step, recognize the movement of the mouth to be photographed in the gesture recognized in the first recognition step , As recognition of the movement of the mouth to be photographed, a second recognition step for recognizing the number of sounds based on the movement of the mouth in the gesture, a semantic candidate corresponding to the result of the gesture recognition by the first recognition step, and a second recognition A determination step for determining the meaning of the gesture based on the recognition result of the movement of the mouth to be imaged in the step; An output step of outputting the meaning of the gesture which is determined at the decision step, the number of sounds that have been recognized by the second recognition step, if exceeded the number of characters meaning it is determined in decision step, the sound speed that has been recognized by the second recognition step And an unused information holding step for holding, as unused information, difference information between the meaning character number determined in the determining step and the determining step includes unused information in the gesture immediately before the gesture to be recognized The meaning of the gesture is determined by adding the unused information to the recognized number of sounds recognized in the second recognition step .

According to the present invention, the gesture of the photographing target is recognized, the movement of the mouth of the photographing target in the gesture is recognized, and the meaning of the gesture is determined based on the recognition result of the gesture and the movement recognition result of the mouth of the photographing target. to decide. The image recognition device recognizes a gesture using the number of sounds. As described above, the image recognition apparatus makes a determination in consideration of the movement of the mouth to be imaged, and thus can accurately recognize the gesture. Furthermore, the image recognition apparatus can correct the synchronization shift even when the movement of the mouth to be imaged at the time of the gesture and the gesture are not synchronized.
The image recognition apparatus of the present invention includes an image acquisition unit that acquires a shooting target as image data, a first recognition unit that recognizes a shooting target gesture based on the image data acquired by the image acquisition unit, Based on the image data acquired by the image acquisition means, the movement of the mouth of the photographing target in the gesture recognized by the first recognition means is recognized, and the movement of the mouth in the gesture is recognized as the recognition of the movement of the mouth of the photographing target. Based on the second recognition means for recognizing the number of sounds based on, the semantic candidates corresponding to the recognition result of the gesture by the first recognition means, and the recognition result of the movement of the mouth to be imaged by the second recognition means. Determination means for determining the meaning of the gesture, and output means for outputting the meaning of the gesture determined by the determination means. The determination means includes the number of characters of the meaning meaning of the gesture, 2 If the difference from the number of sounds recognized by the recognition means is greater than or equal to a predetermined value, the number of sounds recognized by the second recognition means is assumed to be at least one of the gesture meaning candidates, Determine the meaning.
The image recognition method of the present invention is an image recognition method of an image recognition apparatus for recognizing a gesture to be photographed, wherein an image acquisition step for obtaining the photographing target as image data and the image data obtained by the image acquisition step are provided. Based on the first recognition step for recognizing the gesture to be photographed and the image data obtained by the image obtaining step, the movement of the mouth to be photographed in the gesture recognized in the first recognition step is recognized and photographed. As a recognition of the movement of the target mouth, a second recognition step for recognizing the number of sounds based on the movement of the mouth in the gesture, a semantic candidate corresponding to the result of the gesture recognition by the first recognition step, and a second recognition step A judgment step for judging the meaning of the gesture based on the recognition result of the movement of the mouth of the subject to be photographed. An output step for outputting the meaning of the gesture determined in the step, and the determining step includes a case where the difference between the number of characters of the gesture meaning candidate and the number of sounds recognized in the second recognition step is equal to or greater than a predetermined value. The meaning of the gesture is determined by assuming that the number of sounds recognized in the second recognition step is the number of at least one character of the gesture meaning candidates.
According to the present invention, the image recognition apparatus can correct the synchronization shift even when the movement of the mouth to be imaged at the time of the gesture and the gesture are not synchronized.

  Generally, a person who makes a gesture tends to express the meaning of the gesture in the mouth while gesturing. In the present invention, in consideration of the above-mentioned tendency, since the gesture is recognized using the recognition result of the movement of the mouth to be photographed, the meaning of the gesture is judged more accurately than when the gesture is recognized only from the gesture. be able to.

  In the image recognition device of the present invention, the second recognition means may recognize the number of sounds based on the movement of the mouth in the gesture as the recognition of the movement of the mouth to be imaged. In this case, the image recognition apparatus recognizes the gesture using the number of sounds. As described above, the image recognition apparatus makes a determination in consideration of the movement of the mouth to be imaged, and thus can accurately recognize the gesture.

  In the image recognition apparatus of the present invention, the second recognition means is based on the number of times the mouth is opened and closed within a period from the time when the mouth is first opened to the time when the mouth is finally closed during the gesture. The number of sounds may be recognized. In this case, the image recognition apparatus can accurately determine the number of sounds based on the movement of the mouth corresponding to the gesture.

  In the image recognition apparatus of the present invention, the determination unit determines the meaning of the gesture based on the degree of coincidence between the number of sounds of the gesture meaning candidate and the number of sounds in the gesture by the second recognition unit. Also good. In this case, the image recognition apparatus determines the final meaning of the gesture based on the degree of coincidence between the number of sounds of the semantic candidates and the number of sounds in the gesture, and thus can accurately determine the meaning of the gesture.

  In the image recognition apparatus of the present invention, the second recognizing unit recognizes the sound information based on the mouth movement as the mouth movement recognition, and the judging unit recognizes the sound information of the gesture meaning candidate and the second recognition unit. The meaning of the gesture may be determined based on the degree of coincidence with the sound information of the recognition result by the means. In this case, the image recognition apparatus determines the meaning of the gesture based further on the degree of coincidence between the sound information (for example, whether or not it is a consonant) and the gesture candidate. Thereby, the image recognition apparatus compares the sound information generated at the time of the gesture with the sound information of the gesture meaning candidate, and thus can accurately determine the meaning of the gesture.

  Further, in the image recognition device of the present invention, the determination means includes evaluation management information including an evaluation value based on the number of characters for each of the meaning candidates for each character number group distinguished by the number of characters of the plurality of meaning candidates in the gesture. Are generated, and the meaning of the gesture is determined by selecting one meaning candidate from the plurality of meaning candidates using the evaluation value in the evaluation management information and the number of characters distinguished by the character number group. Also good. As described above, the image recognition device uses the evaluation value based on the number of characters for each of the meaning candidates and the number of characters distinguished in the character number group for each character number group distinguished by the number of characters of the plurality of meaning candidates of the gesture. Since the candidate is determined, the synchronization shift can be corrected even when the movement of the mouth to be imaged at the time of the gesture and the gesture are not synchronized.

  According to the present invention, it is possible to accurately determine the meaning of a gesture operation.

It is a block diagram which shows the function structure of the image recognition apparatus 10 of this embodiment. 2 is a hardware configuration diagram of the image recognition device 10. FIG. It is a figure which shows the example of data of gesture DB13 and sentence interpretation DB14. It is explanatory drawing which showed the operation | movement of a signer typically. It is a figure explaining the method of calculating the number of sounds based on the movement of a lip. It is a figure explaining the weighting of the candidate of the meaning of a gesture. It is explanatory drawing which showed the operation | movement of a signer typically. It is a figure explaining the method of calculating the number of sounds based on the movement of a lip. It is a flowchart which shows the translation process of sign language which concerns on embodiment of this invention.

  Hereinafter, this embodiment will be described with reference to the drawings. Where possible, the same parts are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram illustrating functions of the image recognition apparatus 10 according to the present embodiment. The image recognition apparatus 10 is an apparatus for translating a gesture such as a sign language into text or an image, and includes a gesture input unit 11 (image acquisition unit), a gesture recognition unit 12 (first recognition unit), a gesture DB 13, It includes a sentence interpretation DB 14, a lip motion recognition unit 15 (second recognition unit), an unused information DB 17 (unused information holding unit), a determination unit 18 (determination unit), and a determination result output unit 19 (output unit). Has been. The image recognition device 10 is preferably a mobile terminal such as a mobile terminal or a smartphone.

  FIG. 2 is a hardware configuration diagram of the image recognition apparatus 10. As shown in FIG. 2, the image recognition apparatus 10 shown in FIG. 1 physically includes one or a plurality of CPUs 101, a main memory RAM 102 and ROM 103, and input devices 104 such as a keyboard and a mouse. The computer system includes an output device 105 such as a display, a communication module 106 that is a data transmission / reception device such as a network card, an auxiliary storage device 107 such as a semiconductor memory, and the like. Each function in FIG. 1 operates the input device 104, the output device 105, and the communication module 106 under the control of the CPU 101 by loading predetermined computer software on the hardware such as the CPU 101 and the RAM 102 shown in FIG. In addition, it is realized by reading and writing data in the RAM 102 and the auxiliary storage device 107. Hereinafter, each functional block will be described based on the functional blocks shown in FIG.

  The gesture input unit 11 is a part that acquires, as an image, an action of a person to be photographed such as a sign language person, and is a camera, for example.

  The gesture recognition unit 12 is a part that recognizes a gesture based on the action of the person to be photographed specified from the image acquired by the gesture input unit 11. The gesture recognizing unit 12 discriminates the movement of the finger, arm, and upper body of the person to be photographed from a plurality of images, and recognizes the meaning corresponding to the discriminated movement with reference to the gesture DB 13 and the sentence interpretation DB 14.

  The gesture DB 13 is a database describing an operation pattern for causing the gesture recognition unit 12 to recognize a signer's operation. This is a general database for gesture recognition.

  A data example of the gesture DB 13 is shown in FIG. As shown in FIG. 3A, the gesture DB 13 includes “gesture ID”, “hand orientation”, “hand angle”, “finger protrusion presence / absence”, “hand movement information”, and the like. Have

  For example, a gesture whose gesture ID is “00000001” is that the direction of the hand is the front, the angle of the hand is 0 degrees (with the fingertip facing up), and only the thumb is based on “the presence or absence of a finger protrusion” Shows that the hand has stopped, based on "movement".

  The sentence interpretation DB 14 is a database for interpreting the meaning when the gesture recognition unit 12 recognizes a certain amount of motion with reference to the gesture DB 13. Here, an identifier of an operation pattern and its meaning (intention) are described in association with each other. This is also a database for general gesture analysis.

  A data example of the sentence interpretation DB 14 is shown in FIG. As shown in FIG. 3B, the sentence interpretation DB 14 has “gesture ID” and “meaning”.

  In the example of FIG. 3B, a gesture having a gesture ID “00000001” indicates that “a” means a finger character. A gesture with a gesture ID “00002000” indicates that the nouns “style”, “form”, and “type” have meanings. That is, a gesture having a gesture ID “00002000” has a plurality of meanings.

  The lip motion recognition unit 15 recognizes the movement of the lips as the motion of the mouth in the motion unit recognized by the gesture recognition unit 12. Specifically, the lip motion recognition unit 15 recognizes the number of sounds based on the result of measuring the number of lip movements in the motion unit recognized by the gesture recognition unit 12. More specifically, the lip motion recognition unit 15 is based on the result of measuring the number of times the lips are opened and closed during the period from the time when the lips are first opened to the time when the lips are finally closed. To recognize the number of sounds. Also, the lip motion recognition unit 15 recognizes whether a consonant or a vowel is emitted based on the shape of the lips.

  Note that the lip motion recognition unit 15 determines whether or not there is a sound to extend based on the average time of lip motion. For example, as shown in FIG. 4 (A), when the person to be photographed utters “Okinaban”, the lip motion recognition unit 15 performs the “large” lip movement. From the time information of “ki” and “na” instead of three sounds, it is estimated that “o” is two sounds. As shown in FIG. 4B, the lip motion recognizing unit 15 similarly detects the lip of the lip from the average time of the previous and next sound, such as “crow”, which has little lip movement. Analogize the movement.

  The determination unit 18 determines the meaning of the gesture by using the recognition result by the gesture recognition unit 12 and the recognition result by the lip motion recognition unit 15. Specifically, when one recognition result is obtained by the gesture recognition unit 12, the determination unit 18 outputs a meaning indicating the recognition result to the determination result output unit 19. Further, when a plurality of recognition results are obtained by the gesture recognition unit 12, the determination unit 18 narrows down the recognition results using the number of sounds based on the lip movement by the lip motion recognition unit 15.

  Next, a specific example in which the determination unit 18 determines the meaning of the gesture will be described with reference to FIG. FIGS. 5A to 5E are obtained by dividing the sign language action “What hairstyle do you want?”. Based on the result recognized by the gesture recognition unit 12, the motion is divided into FIGS. 5A to 5E, and the lip motion recognition unit 15 recognizes the movement of the lips in the divided unit. As shown in FIGS. 5A to 5E, in the sign language operation “What hairstyle do you want?”, The meaning of the operation indicating “style” in FIG. As shown in FIG. 3 (B), there is also a meaning of “shape” and “type”, so that only the recognition result by the gesture recognition unit 12 cannot narrow down the meaning to one.

  Therefore, the determination unit 18 narrows down the meaning of the gesture using the recognition result by the lip motion recognition unit 15. Specifically, the determination unit 18 narrows down the meaning based on the number of sounds recognized by the lip motion recognition unit 15 and consonant information derived from the lip motion.

  As shown in the sentence interpretation DB 14 shown in FIG. 3B, as a candidate for the “style” part in FIG. 5C, in the sign language action “What hairstyle do you want?” Shown in FIG. The meaning of the action indicating the style (the action with the gesture ID “00002000”) includes “shape” and “type” in addition to “style”. Therefore, there are “style”, “shape”, and “type” as candidates for the meaning of the gesture. In addition, since there is a possibility that “ni” is added to the meaning of the action, “in style”, “in shape”, and “in type” are also included in the candidates. As described above, when the meaning of the action recognized by the gesture recognition unit 12 is determined as a noun, the determination unit 18 also uses a case particle given one sound as a candidate.

  The determination unit 18 first weights each candidate based on the number of sounds recognized by the lip motion recognition unit 15. Specifically, when the number of candidate characters matches the number of sounds, the determination unit 18 sets the number of sounds as the weight of the candidate. Since the lip motion recognition unit 15 has recognized that the number of active sounds corresponding to the “style” is four, the determination unit 18 gives four points to the four-tone candidates. Further, the determination unit 18 sets the weight of the candidate other than the four sounds as a value obtained by subtracting the difference value between the number of candidate sounds and the four sounds from the four points. FIG. 6 shows an example in which each candidate is weighted. Candidate 1's “shape” has 3 sounds, 3 points, candidate 2's “shape” has 2 sounds, and candidate 3's “salt” has 4 sounds, so 4 points Candidate 1a's “To Shape” has 4 sounds, so 4 points, Candidate 2a's “Kani ni” has 3 sounds, and Candidate 3a's “Stunni” has 5 sounds. There are 3 points.

  The determination unit 18 subsequently weights each candidate based on the degree of coincidence with the consonant information recognized by the lip motion recognition unit 15. In this example, the lip motion recognition unit 15 recognizes “st ??”, the first sound is recognized as “s”, the second sound is recognized as “t”, the third sound and the fourth sound. Indicates that the sound could not be recognized.

  Based on the recognition result of the consonant information by the lip motion recognition unit 15, the determination unit 18 determines whether or not each candidate matches the consonant information. If they do not match, 0.5 point is subtracted. . The part that could not be recognized is out of judgment. In the case of the above example, the determination unit 18 cannot recognize the third sound and the fourth sound, so the determination targets are the first sound and the second sound.

  Since the first sound does not match the candidate 1 “shape”, 0.5 points are subtracted to 2.5 points. Candidate 2 “Kata” does not match the first sound, so 0.5 points are subtracted to 1.5 points. The candidate 3 “Sutaru” is not subtracted because the first and second sounds match. Since the first sound does not match the candidate 1a “in shape”, 0.5 points are subtracted to 3.5 points. Candidate 2a “Kani ni” does not match the first sound, so 0.5 points are subtracted to 2.5 points. Candidate 3a “suta ni ni” is not subtracted because the first and second sounds match.

  As described above, the determination unit 18 normalizes the weight calculated based on the degree of coincidence with the number of sounds and the consonant information by dividing by the detected number of sounds (four sounds). As a result, the candidate 3 has the highest weight of “Suru”, and therefore the determination unit 18 sets the candidate 3 as the highest priority candidate, the normalized value corresponding to the candidate 3, and a predefined threshold value. If the compared and normalized value is equal to or greater than the threshold value, the determining unit 18 determines that the candidate 3 has a gesture meaning.

(If the number of sounds from the movement of the lips is different from the number of characters in the meaning of the gesture)
In the example shown in FIG. 5, the case where the gesture and the movement of the lip corresponding to the gesture are synchronized has been described. However, there is a difference between the number of sounds due to the movement of the lip corresponding to the gesture and the gesture. A determination method in a case will be described. With reference to FIG. 7, an example in which the gesture and the movement of the lips corresponding to the gesture are not synchronized in the sign language action “What kind of hair style do you want?” Will be described. As shown in FIG. 7, when there is a divergence between the number of characters in the meaning of the target gesture and the number of lip movements corresponding to the gesture (for example, the maximum number of characters in a gesture and the number of lip movements are two or more divergences). If it is), it is assumed that the movement of the lips corresponding to the next gesture is included, and scoring (weighting) is performed with a plurality of patterns.

  In the case of the example shown in FIG. 7, the operation and the movement of the lips are synchronized during the “what” operation shown in FIG. 7A and the “hair” operation shown in FIG. As shown in FIG. 7 (C), when the movement of the next “Yes” lip is recognized during the “Style” operation, it is determined that the lip movement has occurred six times during the “Style” operation. Thus, there is a divergence between the number of characters in the meaning of the gesture and the number of lip movements (number of sounds) corresponding to the gesture.

  The determination unit 18 compares the number of characters in the meaning of the gesture with the number of sounds for each gesture, and if there is a divergence as a result of the comparison (for example, the difference between the maximum lowercase number of a certain gesture and the number of sounds is 2 or more). Is weighted assuming that the number of sounds of the next gesture is also included, and that the number of sounds is one of the number of characters of the meaning candidate of the gesture. When there are a plurality of gesture meaning candidates recognized by the gesture recognition unit 12, the determination unit 18 performs weighting assuming that the number of sounds is the number of characters of the plurality of meaning candidates.

  As shown in FIG. 7C, when the lips move 6 times, the number of characters in the meaning of the gesture corresponding to the operation corresponding to FIG. 7C (gesture ID “00002000”) is 2 to 4 characters. In this case, assuming that the number of sounds is 2 to 4 characters, the determination unit 18 performs weighting for each case of 2 to 4 characters (divided into semantic candidate character number groups). To do. That is, the determination unit 18 weights the number of sounds recognized by the lip motion recognition unit 15 assuming that the number of characters is at least one of the gesture meaning candidates. Specifically, as shown in FIGS. 8A and 8B, the determination unit 18 determines the weight value of each candidate (value based on the evaluation value and the number of characters) for each semantic candidate character number group. Table-format information including () is generated as evaluation management information.

  The weight value of each candidate and the movement of unused lips when each candidate is applied, assuming that the number of sounds recognized by the lip motion recognition unit 15 in FIG. Indicates the number of times. 8B, assuming that the number of sounds recognized by the lip motion recognition unit 15 is 3 characters, the weight value of each candidate and the unused lips when each candidate is applied. Indicates the number of movements. Here, the unused lip movement refers to a lip movement that was not used to determine the meaning of the gesture. The number of unused lip movements is obtained from a difference value between the number of lip movements and the number of characters of each candidate. Note that the candidates shown in FIGS. 8A and 8B are representative candidates (candidates with high weights). Also, the figure of the weight value of each candidate when the number of sounds recognized by the lip motion recognition unit 15 is assumed to be two characters and the number of unused lip movements when each candidate is applied is omitted. To do.

  The weight shown in FIG. 8A is obtained by normalizing (dividing by the number of characters) the result of weighting based on the number of sounds and weighting based on the consonant information in the case of 4 characters. The weight shown in FIG. 8B is obtained by normalizing the result of weighting based on the number of sounds and weighting based on the consonant information in the case of three characters.

  When it is assumed that the number of sounds recognized by the lip motion recognition unit 15 is 4, 3, or 2 characters (see FIGS. 8A and 8B), the determination unit 18 determines the weight value of each candidate. The highest priority candidate is set as the highest priority candidate, the weight value corresponding to the highest priority candidate is compared with a predefined threshold value, and the weight value corresponding to the highest priority candidate is equal to or greater than the threshold value. The determination unit 18 determines that the highest priority candidate is the meaning of the gesture. This is the end of the description of the specific example in which the determination unit 18 determines the meaning of the gesture.

  Further, the determination unit 18 registers the number of unused lip movements as unused information in the unused information DB 17. The unused information DB 17 is a database that manages unused information. The determination result output unit 19 outputs the determination result by the determination unit 18. In addition, when there are a plurality of candidates as a result of determination by the determination unit 18, the determination result output unit 19 also outputs a candidate for prompting selection.

  Next, a sign language translation process performed by the image recognition apparatus 10 configured as described above will be described. FIG. 9 is a flowchart showing sign language translation processing of the image recognition apparatus 10.

  The sign language action of the sign language to be photographed is photographed by the gesture input unit 11, and an image to be photographed is input (step S1). The photographed sign language motion is recognized by the gesture recognition unit 13 and the lip motion recognition unit 15 recognizes the lip motion (step S2). When the gesture recognition unit 13 recognizes the gesture and the number of candidates for the gesture meaning is only one (step S3; NO), the determination unit 18 recognizes the one recognition result, so that the recognition is completed. (Step S12), the process proceeds to Step S13. When there are a plurality of candidates as a result of gesture recognition by the gesture recognition unit 13 (step S3: YES), the determination unit 18 recognizes that the lip motion recognition unit 15 has a lip motion (step S4; YES). ), Move to step S5. If there is no lip motion as a result of recognition by the lip motion recognition unit 15 (step S4; NO), if there is unused data among the lip motions corresponding to the previous gesture (step S6; YES) ), The process moves to step S5, and if there is no unused data in the lip movement corresponding to the previous gesture (step S6; NO), the determination result output unit 19 outputs a plurality of gesture meaning candidates. The user is prompted to select a gesture meaning candidate (step S11). When the meaning of the gesture is selected, the determination unit 18 determines that the meaning of the selected gesture is the formal meaning of the gesture, and moves to step S13.

  In step S5, the determination unit 18 performs weighting processing on the meaning candidates of each gesture, and if there is a result exceeding the threshold as a result of the weighting processing (step S7; YES), the recognition is completed (step S9). .

  In step S5, if there is no candidate that exceeds the threshold (step S7; NO) as a result of the weighting process of the semantic candidates of each gesture by the determination unit 18, the determination result output unit 19 outputs a plurality of candidates, The user is prompted to select a meaning candidate (step S8). The determination unit 18 determines that the meaning of the selected gesture is the official meaning of the gesture. After step S8 or step S9, if there is unused data, an unused data registration process is performed (step S10), and the process proceeds to step S13. Here, the unused data registration process is a process in which the determination unit 18 registers unused data in the unused information DB 17.

  In step S13, it is determined whether or not the recognition has been completed for all the gestures input by the gesture input unit 11 in step S1, and if the recognition has not been completed (step S13; NO), the process returns to step S2. Recognition of the recognized gesture and recognition of the lip motion corresponding to the gesture are performed (step S2).

  In step S13, when the recognition is completed for all the gestures input by the gesture input unit 11 in step S1 (step S13; YES), the determination result output unit 19 outputs the recognition result by the determination unit 18 ( Step S14) and the process is terminated.

  Next, functions and effects of the image recognition device 10 according to the present embodiment will be described.

  According to the image recognition device 10 of the present embodiment, the gesture recognition unit 12 recognizes a shooting target gesture from the image data input by the gesture input unit 11, and the lip motion recognition unit 15 inputs by the gesture input unit 11. The determination unit 18 determines the meaning of the gesture based on the meaning candidate corresponding to the gesture recognition result and the recognition result of the movement of the lip of the shooting target. The determination result output unit 19 outputs the meaning of the gesture.

  In this way, the image recognition apparatus 10 recognizes a gesture using not only the gesture recognition result but also the recognition result of the movement of the lip of the photographing target in the gesture. Generally, a person who makes a gesture tends to express the meaning of the gesture in the mouth while gesturing. In consideration of this point, the image recognition apparatus 10 recognizes the gesture using the recognition result of the movement of the lip to be photographed. Therefore, the meaning of the gesture is more accurately compared to the case of recognizing only from the gesture. Judgment can be made.

  The lip motion recognizing unit 15 recognizes the number of sounds based on the movement of the lip to be imaged during the gesture as the mouth motion recognition. In this case, the image recognition apparatus 10 recognizes a gesture using the number of sounds to be captured. As a result, the image recognition apparatus 10 judges the meaning of the gesture by comparing the number of sounds of the gesture and the number of sounds based on the movement of the lips to be photographed, so that the gesture can be recognized accurately. .

  The lip motion recognition unit 15 recognizes the number of sounds based on the number of times the lips are opened and closed within the period from the time when the lips are first opened to the time when the lips are finally closed. In this case, the image recognition device 10 can accurately determine the number of sounds based on the lip motion corresponding to the gesture.

  The determination unit 18 determines the meaning of the gesture based on the degree of coincidence between the number of sounds of the gesture meaning candidate and the number of sounds in the gesture by the lip motion recognition unit 15. In this case, since the image recognition apparatus determines the meaning of the gesture based on the degree of coincidence between the number of sounds of the gesture meaning candidate and the number of times the lips are opened and closed, it is possible to accurately determine the meaning of the gesture.

  The lip motion recognition unit 15 recognizes the sound information based on the lip motion as the mouth motion recognition, and the determination unit 18 determines the sound information of the gesture meaning candidate and the sound information of the recognition result by the second recognition unit. The meaning of the gesture is determined based on the degree of coincidence. In this case, the image recognition apparatus 10 determines the meaning of the gesture based on the degree of coincidence regarding the sound information between the sound information (for example, whether or not it is a consonant) and the meaning candidate of the gesture. As a result, the image recognition apparatus 10 compares the sound information generated at the time of the gesture with the sound information of the gesture meaning candidate, and thus can accurately determine the meaning of the gesture.

  If the number of sounds recognized by the lip motion recognition unit 15 exceeds the number of characters determined by the determination unit 18, the number of sounds recognized by the lip motion recognition unit 15 and the number of characters determined by the determination unit 18 The unused information DB 17 that holds the difference information as unused information is further included, and when there is unused information in the gesture immediately before the gesture to be recognized, the determination unit 18 uses the unused information as the lip motion recognition unit. In addition to the number of sounds recognized by 15, the meaning of the gesture is determined. In this case, the image recognition apparatus 10 can recognize the gesture by correcting the deviation even when the gesture and the movement of the lip of the photographing target are not synchronized.

  When the difference between the number of characters of the gesture meaning candidates and the number of lip motions recognized by the lip motion recognition unit 15 is equal to or greater than a predetermined value, the determination unit 18 determines the number of sounds recognized by the lip motion recognition unit 15 as the gesture. Any one of the meaning candidates is assumed to be at least one character number. In this case, the image recognition apparatus 10 can recognize the gesture by correcting the deviation even when the number of sounds generated by the movement of the lips to be imaged at the time of the gesture and the meaning of the gesture are not synchronized.

  When there are a plurality of gesture meaning candidates and the number of characters in each gesture meaning is not the same, the determination unit 18 classifies the number of sounds recognized by the lip motion recognition unit 15 by the number of characters in the plurality of gesture meaning candidates. To determine the meaning of the gesture. More specifically, the determination unit 18 generates a plurality of pieces of evaluation management information including evaluation values based on the number of characters for each of the semantic candidates for each character number group distinguished by the number of characters of the plurality of semantic candidates in the gesture. Then, the meaning of the gesture is determined by selecting one meaning candidate from a plurality of meaning candidates using the evaluation value in the evaluation management information and the number of characters distinguished in the character number group. In this case, the image recognition device 10 makes sense based on the evaluation value based on the number of characters for each semantic candidate and the number of characters distinguished in the character number group for each character number group distinguished by the number of characters of the plurality of semantic candidates for the gesture. Since the candidate is determined, the gesture can be recognized by correcting the deviation even when the number of sounds generated by the movement of the lip to be imaged at the time of the gesture and the meaning of the gesture are not synchronized.

  In the above-described embodiment, the case where a plurality of semantic candidates are extracted from the same operation has been described. However, a plurality of semantic candidates including a similar operation may be extracted.

  In the above-described embodiment, when the meaning of the previous gesture cannot be specified, a candidate for the meaning of the gesture is output to prompt selection, but the determination of the meaning of the previous gesture is skipped, and the meaning of the subsequent gesture May be specified, and the meaning of the previous gesture skipped thereafter may be specified.

  In the above-described embodiment, the case where the lip motion recognition unit 15 recognizes the lip motion has been described as the mouth motion. However, the tongue motion may also be recognized as the mouth motion. In this case, it is recognized whether a consonant or a vowel is emitted based on the movement of the lips and the movement of the tongue.

DESCRIPTION OF SYMBOLS 10 ... Image recognition apparatus, 11 ... Gesture input part, 12 ... Gesture recognition part, 13 ... Gesture DB, 14 ... Text interpretation DB, 15 ... Lip motion recognition part, 17 ... Unused information DB, 18 ... Judgment part, 19 ... Judgment result output section.

Claims (8)

An image acquisition means for acquiring a shooting target as image data;
First recognition means for recognizing a gesture to be imaged based on image data acquired by the image acquisition means;
Based on the image data acquired by the image acquisition unit, the movement of the mouth of the photographing target in the gesture recognized by the first recognition unit is recognized, and the movement of the mouth of the shooting target is recognized as the movement of the mouth of the shooting target. Second recognition means for recognizing the number of sounds based on the movement of the mouth in
Determination means for determining the meaning of a gesture based on a meaning candidate corresponding to a gesture recognition result by the first recognition means and a recognition result of the mouth movement of the photographing target by the second recognition means;
Output means for outputting the meaning of the gesture determined by the determination means;
When the number of sounds recognized by the second recognizing means exceeds the number of characters determined by the determining means, the number of sounds recognized by the second recognizing means and the number of characters determined by the determining means Unused information holding means for holding the difference information as unused information,
Prepared ,
If there is unused information in the gesture immediately before the gesture to be recognized, the determining means determines the meaning of the gesture by adding the unused information to the recognized number of sounds recognized by the second recognizing means. An image recognition device. An image acquisition means for acquiring a shooting target as image data;
First recognition means for recognizing a gesture to be imaged based on image data acquired by the image acquisition means;
Based on the image data acquired by the image acquisition unit, the movement of the mouth of the photographing target in the gesture recognized by the first recognition unit is recognized, and the movement of the mouth of the shooting target is recognized as the movement of the mouth of the shooting target. Second recognition means for recognizing the number of sounds based on the movement of the mouth in
Determination means for determining the meaning of a gesture based on a meaning candidate corresponding to a gesture recognition result by the first recognition means and a recognition result of the mouth movement of the photographing target by the second recognition means;
Output means for outputting the meaning of the gesture determined by the determination means;
Equipped with a,
When the difference between the number of characters of the semantic meaning of the gesture and the number of sounds recognized by the second recognition unit is equal to or greater than a predetermined value, the determination unit determines the number of sounds recognized by the second recognition unit as An image recognition apparatus that determines the meaning of a gesture on the assumption that the number of characters is at least one of the gesture meaning candidates . The second recognition unit recognizes during the gesture, the first time you open the mouth, in the period until the last time mouth closed, the number of notes based on the number of opening and closing of the port, wherein Item 3. The image recognition apparatus according to Item 1 or 2 . The determining means includes a number of notes meanings candidate gesture to determine the meaning of the gesture based on the coincidence degree between the sound speed in the in the gesture by the second recognizing means, any one of claims 1 to 3 The image recognition apparatus described in 1.   The second recognizing means recognizes sound information by mouth movement as recognition of the mouth movement, and the judging means recognizes sound information of the meaning meaning of the gesture and the recognition result by the second recognizing means. The image recognition device according to claim 4, wherein the meaning of the gesture is further determined based on the degree of coincidence with the sound information. The determination means generates a plurality of evaluation management information including an evaluation value based on the number of characters for each of the meaning candidates for each character number group distinguished by the number of characters of each of the plurality of meaning candidates in the gesture,
By using the evaluation value in the evaluation management information, a number of characters are distinguished by the characters group, by selecting one of the meanings candidates from a plurality of sense candidates, to determine the meaning of the gesture, according to claim 2 Image recognition device. In an image recognition method of an image recognition apparatus for recognizing a gesture to be photographed ,
An image acquisition step of acquiring a shooting target as image data;
A first recognition step for recognizing a gesture to be imaged based on the image data acquired by the image acquisition step;
Based on the image data acquired in the image acquisition step, the movement of the mouth of the shooting target in the gesture recognized in the first recognition step is recognized, and the movement of the mouth of the shooting target is recognized as the movement of the mouth of the shooting target. A second recognition step for recognizing the number of sounds based on the mouth movement in
A determination step of determining the meaning of the gesture based on the meaning candidates corresponding to the gesture recognition result in the first recognition step and the recognition result of the mouth movement of the photographing target in the second recognition step;
An output step of outputting the meaning of the gesture determined in the determination step ;
If the number of sounds recognized in the second recognition step exceeds the number of characters determined in the determination step, the number of sounds recognized in the second recognition step and the number of characters determined in the determination step Unused information holding step for holding the difference information as unused information,
Prepared,
In the determination step, when there is unused information in the gesture immediately before the gesture to be recognized, the meaning of the gesture is determined by adding the unused information to the recognized number of sounds recognized in the second recognition step. An image recognition method. In an image recognition method of an image recognition apparatus for recognizing a gesture to be photographed,
  An image acquisition step of acquiring a shooting target as image data;
  A first recognition step for recognizing a gesture to be imaged based on the image data acquired by the image acquisition step;
  Based on the image data acquired in the image acquisition step, the movement of the mouth of the shooting target in the gesture recognized in the first recognition step is recognized, and the movement of the mouth of the shooting target is recognized as the movement of the mouth of the shooting target. A second recognition step for recognizing the number of sounds based on the mouth movement in
  A determination step of determining the meaning of the gesture based on the meaning candidates corresponding to the gesture recognition result in the first recognition step and the recognition result of the mouth movement of the photographing target in the second recognition step;
  An output step of outputting the meaning of the gesture determined in the determination step;
With
  When the difference between the number of characters of the meaning meaning of the gesture and the number of sounds recognized by the second recognition step is equal to or greater than a predetermined value, the determining step determines the number of sounds recognized by the second recognition step as An image recognition method for determining the meaning of a gesture on the assumption that the number of characters in any one of gesture meaning candidates is at least one.

Images