JP5534517B2 - Utterance learning support device and program thereof - Google Patents

Description

  The present invention is mainly used for language utterance learning, etc., and compares the lip movement of a speaker when speaking with an exemplary lip movement, and indicates an improvement point of the speaker's lip movement The present invention relates to a learning support apparatus and its program.

Traditionally, in Japan, in order to acquire correct pronunciation in language utterance learning, the speaker can simulate the movements of the lips and tongue for each vowel written in the textbook and at the same time listen to the instructor's pronunciation with his ears. There has been a widespread learning method in which the pronunciation of the teacher is approximated by repeating the pronunciation in front of the leader.
In addition, when a narrator learns pronunciation, he / she cannot hear his / her voice, so he repeats pronunciation by simulating the movement of the lips and tongue in front of the instructor, and the instructor listens to the pronunciation and advises on improvements. By doing so, the learning method of getting closer to the pronunciation of the instructor has been widely practiced.

  However, according to the learning method described above, when a speaker wants to check whether his / her pronunciation is correct, the speaker himself / herself makes a judgment by recording the voice at the time of pronunciation on a tape and listening to it later. However, it was difficult for the speaker to accurately determine his / her proficiency level. In addition, since the narrator cannot hear his own voice, it was impossible to objectively determine how to change the lip movement to improve pronunciation.

  In view of this, as a learning support system for learning correct pronunciation, the speech data when the speaker pronounces the word and the speech data when the instructor pronounces the same word are compared. Something has been developed for scoring or evaluating a person's pronunciation and instructing improvement points.

For example, Patent Literature 1 discloses a learning server device that provides a language learning lesson to a speaker, and analyzes the speech of the speaker's language learning transmitted from a PC (Personal Computer) to generate the pronunciation of the speaker. Techniques for scoring are disclosed.
Further, for example, in Patent Document 2, a user's voice is acquired, and the model voice stored in advance is compared with the acquired user's voice, and the difference between the model voice and the user's voice is based on the comparison result. A language learning device that extracts points, generates emphasis instruction data that indicates an aspect to be emphasized for a portion where the extracted difference exists, and outputs an exemplary voice according to the aspect based on the generated emphasis instruction data It is disclosed.
Further, for example, Patent Document 3 stores one or more instructor data, which is data concatenated for each phoneme, and accepts an input of a speaker's voice, and divides the voice into frames. A pronunciation rating device is disclosed that acquires one or more data, evaluates the voice of a speaker based on instructor data and one or more voice data for each frame, and outputs the rating result.

JP 2007-212558 A JP 2007-139868 A JP 2006-227587 A

On the other hand, in English and American English-speaking countries, it has been considered important to acquire correct lip movements in order to acquire correct pronunciation in language learning.
For this reason, in English-speaking English-speaking countries, the lip movement when the instructor pronounces the word is shown to the speaker, the lip movement of the instructor is imitated to the speaker, and the lip movement at that time is instructed. It is effective to make the speaker's lip movement closer to the instructor's lip movement by evaluating it compared to the lip movement of the speaker and advising improvement points, so that the speaker can learn correct pronunciation. Is established as an effective teaching method.

  However, in the conventional utterance learning support systems described in Patent Documents 1 to 3, since the speech when the speaker utters is analyzed to evaluate the pronunciation of the speaker, the improvement of the lip movement of the speaker Could not be directed. For this reason, it has been desired to establish a technology that enables a speaker to objectively recognize the improvement of his / her lip movement.

  The present invention has been made to solve the above-described problems of the prior art, and based on the result of comparing the lip movement of the speaker with the lip movement of the instructor, the improvement of the lip movement of the speaker is objectively evaluated. It is an object to provide an utterance learning support device and a program thereof that can be shown automatically.

  In order to solve the above-mentioned problem, the invention according to claim 1 is based on an image of a lip portion when a speaker speaks a predetermined word in a certain language photographed by photographing means. Utterance learning support that shows the improvement of the lip movement of the speaker based on the result of comparing this lip movement with the exemplary lip movement when the instructor utters the word An apparatus, comprising: a data storage means, an utterance content designation means, an image processing means, an action measurement means, a data conversion means, a difference calculation means, a correction amount calculation means, and a correction information output means. The configuration.

According to such a configuration, the utterance learning support device stores a plurality of correspondences between at least the word that the utterer speaks and the image of the lip portion when the instructor utters the word by the data storage unit.
Further, the utterance learning support device is configured such that an utterance content designation unit is configured to input a word to be spoken by a speaker from an external input or from a plurality of the words stored in the data storage unit. Designate one by order and instruct the speaker to utter the word. The utterance content designation means, for example, reads a word to be uttered by the speaker from the data storage means, and displays the word on the display device, thereby instructing the utterer to utter the word.

  The utterance learning support device uses the image processing means as a reference for specifying the lip movement of the speaker from the image of the lip portion when the speaker utters the word specified by the utterance content specifying means. A plurality of predetermined feature point positions are extracted.

Furthermore, the utterance learning support device measures the change in position for each of the feature points extracted by the image processing means by the action measuring means as an action history that is a history of lip movement.
Then, the speech learning support device numerically analyzes the motion history for each of the feature points measured by the motion measurement unit by the data conversion unit, so that a plurality of preset spectral components are obtained for each feature point. Convert to the represented operating spectrum.

  Then, the utterance learning support device uses the difference calculation means to obtain the model for each feature point obtained in advance from the motion spectrum for each feature point obtained by the data conversion means and the image of the model lip portion. The difference from the operation spectrum is calculated.

Furthermore, the utterance learning support device compares the absolute value of the difference calculated by the difference calculation unit with a predetermined threshold value for each feature point by the correction amount calculation unit, and the absolute value of the difference is If there is a feature point that is larger than a predetermined threshold, a correction amount that specifies the direction and size of correcting the operation of the feature point is calculated by a predetermined correction function.
Then, the speech learning support system, the correction information output means outputs the correction information corresponding to the correction amount calculated in the correction calculation means Viewing device.

  According to this, the lip movement of the speaker is compared with the lip movement of the instructor, the correction amount of the lip movement of the speaker is obtained, and the correction information corresponding to the correction amount is displayed on the display device. It is possible to make the person objectively recognize the improvement of the lip movement.

  The utterance learning support device according to claim 2 is the utterance learning support device according to claim 1, wherein the correction information output means is the correction amount calculation means on the image of the lip portion of the speaker. An image indicating a direction and a size for correcting the operation of the feature point is synthesized at a position corresponding to the feature point for which the correction amount is calculated, and is output to the display device.

For example, a CG (Computer Graphics) of a graphic (for example, an arrow) that specifies the direction and size to be corrected is synthesized and displayed at the position corresponding to the feature point for correcting the movement on the lip image of the speaker. can do. The CG of this figure may be stored in advance in an appropriate storage unit, or the correction information output unit may generate it each time based on the correction amount calculated by the correction amount calculation unit.
This makes it easier for the speaker to intuitively recognize how to correct his / her lip movement.

The utterance learning support device according to claim 3 is the utterance learning support device according to claim 1 or 2, wherein the correction information output means calculates the correction amount by the correction amount calculation means. A text specifying the feature point and a direction and size for correcting the operation of the feature point is output to the display device.
For example, the text may be synthesized by speech and reproduced by a speaker of the display device, or the text may be displayed on the display screen of the display device.
According to this, since it is displayed in text that how much the action of which feature point should be corrected, it becomes easier for the speaker to understand the improvement of his / her lip movement.

  Further, the utterance learning support program according to claim 4, based on the image of the lip portion of the utterance when the utterer utters a predetermined word in a certain language, which is photographed by the photographing means, In order to show the improvement of the speaker's lip movement based on the result of comparing the lip movement with the exemplary lip movement when the instructor spoke the word , Speech instruction means, image processing means, motion measurement means, data conversion means, difference calculation means, correction amount calculation means, and correction information output means.

  According to such a configuration, the utterance learning support program stores, by the utterance instruction means, at least a plurality of correspondences between the word that the utterer speaks and the image of the lip portion when the instructor utters the word. From the plurality of words stored in the storage means, one word to be spoken by the speaker is selected from the outside or in a preset order, and the speaker is instructed to speak the word. To do.

  The utterance learning support program is set in advance as a reference for specifying the lip movement of the speaker from the image of the lip portion when the speaker utters the word instructed by the utterance instruction unit by the image processing unit. A plurality of extracted feature point positions are extracted.

  The utterance learning support program measures the change in position for each of the feature points extracted by the image processing means as an action history that is a history of lip movements by the action measuring means.

  The utterance learning support program is represented by a plurality of preset spectral components for each feature point by numerically analyzing the motion history for each feature point measured by the motion measurement unit by the data conversion unit. To the operating spectrum.

  The utterance learning support program is represented by a plurality of preset spectral components for each feature point by numerically analyzing the motion history for each feature point measured by the motion measurement unit by the difference calculation unit. To the operating spectrum.

  The utterance learning support program compares the absolute value of the difference calculated by the difference calculating unit with a predetermined threshold by the correction amount calculating unit for each feature point, and the absolute value of the difference is determined in advance. If there is a feature point that is larger than the threshold value, a correction amount that specifies the direction and size of correcting the operation of the feature point is calculated by a predetermined correction function.

Then, the speech learning support program, the correction information output means outputs the correction information corresponding to the correction amount calculated in the correction calculation means Viewing device.

The utterance learning support device and the utterance learning support program according to the present invention have the following excellent effects.
According to the first and fourth aspects of the present invention, the correction amount is calculated based on the result of comparing the lip movement of the speaker with the exemplary lip movement, and the correction information corresponding to the correction amount is displayed on the display device. By doing so, it is possible to objectively show the improvement point of the own lip movement to the speaker, so that correct pronunciation can be effectively learned.
According to the second and third aspects of the invention, it becomes easier for the speaker to understand the correction point of the lip movement.

It is a figure for demonstrating a mode that a speaker learns using the speech learning assistance apparatus in this invention, the example of a screen displayed on a display screen before a learning start, and the screen displayed on a display screen after a learning start An example is shown. It is a block block diagram which shows the structure of the speech learning assistance apparatus of this invention. It is the figure which represented notionally the example of the data structure which the data storage means in the speech learning assistance apparatus of this invention memorize | stores. It is the figure which showed the feature point extracted by the image processing means in the speech learning assistance apparatus of this invention. It is the figure which showed the example of the action history graph produced | generated by the action measurement means in the speech learning assistance apparatus of this invention. It is the figure which showed the example of the operation | movement spectrum graph of the feature point produced | generated by the data conversion means in the speech learning assistance apparatus of this invention. It is the figure which showed the example of the operation | movement spectrum graph of the feature point at the time of uttering the word which has a vowel string close | similar to a Japanese vowel among English words. It is the figure which showed the comparative example of the operation | movement spectrum graph of the feature point between the English vowels which have a vowel string close | similar to a Japanese vowel among English words. (A) shows the ratio of spectral components in the motion spectrum of feature points when a teacher utters a word containing a certain English vowel, and (b) shows the initial time when the speaker (learner) started learning Shows the ratio of spectral components in the operating spectrum of the feature points when the word containing the same English vowel as in (a) is uttered, and (c) shows the same speaker (learner) as in (b) learning. It is a figure which shows the ratio of the spectrum component in the operation | movement spectrum of the feature point when the word containing the same English vowel as (a) and (b) is uttered in the stage advanced to some extent. It is a conceptual diagram for demonstrating the correction function used with the correction amount calculation means in the speech learning assistance apparatus of this invention. It is a figure which shows the example of a screen at the time of displaying the correction point of a speaker's lip movement by the correction information output means in the speech learning assistance apparatus of this invention. It is a flowchart which shows operation | movement of the speech learning assistance apparatus of this invention. It is a block block diagram which shows the structure of the speech learning assistance apparatus which concerns on the modification of embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Outline of speech learning support device]
First, with reference to FIG. 1, the outline | summary of the speech learning assistance apparatus 1 in this invention is demonstrated.
The utterance learning support device 1 includes a lip movement when a speaker utters a word specified in a language to be learned, a teacher who teaches learning of the language, a native speaker whose language is a native speaker, etc. Then, it is simply referred to as an instructor.) Based on the result of comparing the exemplary lip movement when the same word is uttered in advance, the improvement point of the lip movement of the speaker is shown.

  As shown on the right side of FIG. 1, the utterance learning support device 1 includes text data Ta that instructs a word to be uttered by a speaker, and an image TG of an exemplary lip portion when the instructor utters the word. It is displayed on the display screen 31 of the display device 30. This image TG is a moving image that is stored in advance in the utterance learning support device 1 and shows a series of operations of the lip portion of the instructor when the instructor speaks a word to be spoken by the speaker.

  Further, as shown in the left side of FIG. 1, the utterance learning support device 1 displays a screen including, for example, a language selection button, a difficulty level selection button, a learning start button, and the like on the display screen 31 as an initial screen. Yes. The initial screen is stored in advance in a storage unit (not shown) of the utterance learning support device 1 and is displayed on the display screen 31 of the display device 30 when the speaker activates the utterance learning support system. ing. On the other hand, when the display device 30 receives an input of selection of a language to be learned, selection of difficulty level, and determination of learning start by a speaker or the like via, for example, a mouse or the like, a signal indicating the input is received. 1 is output. In this initial screen, the vowel selection screen may be transitioned to after the selection of the language to be learned by the language selection button or the like is input by the speaker. According to this, since the vowel that the speaker is not good at can be learned intensively, convenience is enhanced.

  Then, the speech learning support device 1 receives, from the display device 30, for example, an input of a signal indicating that “English” is selected as the learning language by the speaker and “normal” is selected as the difficulty level. As illustrated on the right side of FIG. 1, the text “Please say“ aunt ”” that indicates the word to be spoken to the speaker, and the lip portion of the lip when the instructor speaks “aunt” are stored in advance. The image Gt is displayed on the display screen 31.

  Then, when the speaker utters the word specified by the utterance learning support device 1 and displayed on the display screen 31 with reference to the exemplary lip movement image TG displayed on the display screen 31, the photographing unit 20. Thus, the lip portion of the speaker is continuously photographed from the start of the utterance to the end of the utterance.

  Then, the utterance learning support device 1 acquires and analyzes the image of the lip portion of the speaker photographed by the photographing unit 20, and the analysis result and the image TG of the lip portion when the instructor utters the same word. The amount of correction of the lip movement of the speaker is calculated based on the result of comparison with the analysis result of the above, and the correction information corresponding to the amount of correction is displayed on the display screen 31 of the display device 30.

  As described above, according to the utterance learning support device 1, the correction amount of the lip movement of the speaker is calculated, and the correction information corresponding to the correction amount is displayed on the display screen 31 of the display device 30. On the other hand, it is possible to objectively show improvement points of lip movement.

  The moving image may be set in advance so as to be repeatedly reproduced a predetermined number of times so that the speaker can repeatedly confirm the exemplary lip movement, or when a signal is input from the outside by the speaker. You may make it reproduce repeatedly.

[Configuration of speech learning support device]
Next, the configuration of the utterance learning support device 1 according to the present invention will be described with reference to FIG.
As shown in FIG. 2, the utterance learning support system includes an utterance learning support device 1, a photographing unit 20, and a display device 30.
Here, prior to the description of the configuration of the utterance learning support device 1, the photographing unit 20 and the display device 30 will be described with reference to FIG.

  The photographing means 20 is for photographing the lip portion of the speaker when the speaker is speaking. The lip image captured by the photographing means 20 is output to the utterance learning support device 1. The photographing unit 20 may be, for example, a general camera or a stereo camera that can detect the displacement of the depth of the lip. Note that the image of the lip is continuously photographed by the photographing means 20 while the speaker is speaking. Further, the lip portion may include not only the periphery of the lip but also the lower part of the speaker's face (from the nose to the lower jaw). When the image processing unit 13 described later performs image processing using an image of the entire face of the speaker, the photographing unit 20 may photograph the entire face of the speaker. The photographing means 20 may be built in the display device 30 as shown in FIG.

  The display device 30 accepts input of text data for instructing the utterance of a word from the utterance learning support device 1 and an image of an exemplary lip movement when the instructor utters the word and displays it to the speaker. And has a display screen 31. For example, the display device 30 can be a PC (Personal Computer), and the display screen 31 can be a PC monitor or the like. In FIG. 1, for convenience of illustration, a scene of an exemplary lip movement is displayed on the display screen 31, but in actuality, the exemplary lip movement is displayed as a moving image. The display device 30 may further include a speaker (not shown).

  The display screen 31 is not limited to displaying an image in a planar manner, and a display screen 31 may be used. When the display screen 31 can display an image three-dimensionally, a moving image of an exemplary lip movement may be displayed three-dimensionally. The three-dimensional display is suitable for learning words including vowels that are pronounced by protruding lips such as “U, u” and “O, o”.

  The utterance learning support device 1 obtains the lip movement of the utterer from the image of the lip portion when the utterer utters a word specified in advance in a certain language, which is photographed by the photographing means 20. Based on the result of comparing the lip movement with the typical lip movement possessed in advance, the improvement of the lip movement of the speaker is shown. The utterance learning support device 1 includes a data storage unit 11, an utterance content designation unit 12, an image processing unit 13, an action measurement unit 14, a data conversion unit 15, a difference calculation unit 16, and a correction amount calculation unit 17. And correction information output means 18. Here, the utterance learning support device 1 does not input the voice when the speaker utters, and obtains the lip movement of the speaker from only the moving image captured by the imaging means 20.

  The data storage means 11 stores a plurality of correspondences between at least a word to be spoken by a speaker and an image of a lip portion when the instructor utters the word, such as a nonvolatile memory (NVRAM), a hard disk, etc. This is a general storage medium.

Here, an example of the data structure in the data storage unit 11 will be described with reference to FIG. In FIG. 3, a part of the data structure in the data storage means 11 is conceptually represented.
As shown in FIG. 3, the data storage means 11 includes a word existing in each language, text data (text 1, text 2,..., Text n) that instructs the speaker to speak the word, Exemplary motion spectrum (spectrum 1) for each feature point obtained from the image of the lip portion (image 1, image 2,..., Image n) when the instructor utters the word and the image of the instructor's lip portion. , Spectrum 2,..., Spectrum n), the ratio of the first component of the motion spectrum to the entire motion spectrum (ratio 1, ratio 2,..., Ratio n), and the difficulty of pronunciation of words (difficult, normal, And a plurality of data sets (data 1, data 2,..., Data n) classified into each vowel of the language. The operating spectrum and ratio will be described later in detail.

  Since it is sufficient that at least one word is stored for each vowel, it is sufficient for the data storage means 11 to store at least one data set for each vowel. However, as shown in FIG. It is preferable to store a plurality of data sets for each vowel. If multiple data sets are stored for each vowel, the speaker can speak and learn various words for one vowel, which can improve the learning efficiency of the speaker. It is. Although FIG. 3 shows an example in which two data sets are stored for one vowel, the present invention is not limited to this, and an arbitrary number can be used.

  For example, when the language is Japanese, the vowels are five tones “A, I, U, D, E”, so the data storage means 11 includes a predetermined Japanese word including the vowel “A”. A word, a word including the vowel “I”,..., And a word including the vowel “O” are classified and stored. In addition, the data storage unit 11 stores a data set for each vowel existing in the language, such as a long vowel, a short vowel, a composite vowel, a semi-vowel, and a weak vowel.

  Note that the data storage means 11 may store a plurality of data sets arranged in advance in a certain order (for example, in order of difficulty of pronunciation) within one vowel. Note that the data structure of FIG. 3 is an example, and is not limited to this. For example, the degree of difficulty may not be stored. In this way, each data of the plurality of data sets stored in the data storage unit 11 is appropriately read out by the utterance content designation unit 12 or the difference calculation unit 16.

The utterance content designating unit 12 designates one word to be uttered by the utterer from a plurality of data sets stored in the data storage unit 11 by an external input or in a preset order. Then, the speaker is instructed to utter the word.
Here, the utterance content specifying means 12 also has a function of controlling the display content displayed on the display screen 31 of the display device 30. Here, the utterance content specifying means 12 reads out an initial screen from a storage means (not shown) and displays it on the display screen 31, and a signal indicating that various items are selected by the speaker on the initial screen from the display device 30. When an input of a signal indicating the start of learning is received, one word to be spoken by the speaker is selected from a plurality of data sets stored for each vowel in the data storage unit 11, and the utterance of the word is instructed The text data to be read and the image of the lip when the instructor utters the word in advance are read out and displayed on the display screen 31.

  The image of the lip when the instructor utters the word to be spoken by the speaker in advance shows the exemplary lip movement of the instructor. By displaying this image on the display screen 31, the speaker can Since the user can speak while referring to the exemplary lip movement, the learning efficiency of the speaker can be improved.

  When designating a word to be uttered by a speaker, the utterance content designation unit 12 may randomly select a word from a plurality of data sets stored for each vowel in the data storage unit 11. For example, when the data structure shown in FIG. 3 is stored in the data storage unit 11, the utterance content designation unit 12 first sets the word to be spoken by the speaker to “aunt”, then “cup”, etc. Can be selected at random.

  However, when the difficulty level is designated by the speaker on the initial screen, the utterance content designation unit 12 refers to the difficulty levels in the plurality of data sets stored in the data storage unit 11 and is designated by the speaker. Select words that are suitable for the difficulty level. In the initial screen, when a vowel is specified by the speaker, the utterance content specifying unit 12 selects a word from a plurality of data sets stored in the data storage unit 11 in association with the vowel specified by the speaker. Select as appropriate.

The image processing means 13 extracts the position of a preset feature point in the lip part from the moving image of the lip part photographed by the photographing means 20.
Here, feature points will be described with reference to FIG. As shown in FIG. 4, the feature points are the upper end of the lip part (midpoints of the upper end point of the upper lip ( ₂ midpoints of a ₁ , a ₂ )) A point, lower end B point, left end C point, right end D point A point or 5 points further including the apex of the lower jaw can be used. The four points described above are pixels (pixels serving as boundaries) having a maximum difference in pixel value from pixels having red pixel values indicating the lip portion and pixels having skin color pixel values indicating the face portion. The pixel located at the extreme end coordinates of the upper lip (upper lip) and the lower lip (lower lip) is the object. Further, when extracting the vertex of the lower jaw, a pixel having a maximum pixel value difference (luminance difference) between a pixel having a skin color pixel value of the jaw and a pixel having a skin color pixel value of the neck, and What is necessary is just to extract the pixel located in the lowest end.

  As long as the moving image of the lip portion is continuously input from the photographing unit 20, the image processing unit 13 continues to extract extraction points for each image. That is, the image processing means 13 extracts four or five points for one image, extracts each of a plurality of images (usually 30 frames per second) constituting the moving image, and extracts the feature points. Are output as time-series data along the time when the moving image of the lip portion was captured. The image processing means 13 outputs the extracted feature point positions to the motion measuring means 14.

  Although the image processing unit 13 extracts four or five feature points here, the present invention is not limited to this, and feature points with an arbitrary number (six or more) may be extracted.

The motion measuring unit 14 measures the change in the position of the feature point with respect to the position of the feature point extracted by the image processing unit 13 as a motion history that is a history of lip motion.
Then, the motion measuring unit 14 measures an operation history representing the change of each feature point for each of the four or five feature points extracted by the image processing unit 13. When the moving image of the lip portion is continuously input to the image processing means 13 and the feature point is extracted, if there is no change in the feature point for a certain time (for example, 2 seconds), that is, the moving image of the lip portion. If there is no change in the image (no movement), the motion measuring means 14 determines that the speaker is not speaking.

  Here, an example of an operation history (operation history graph) will be described with reference to FIG. The movement history graph shown in FIG. 5 shows changes in point B (see FIG. 4) of the lower lip (lower lip), with the horizontal axis representing time t (ms) and the vertical axis representing displacement y (mm). ). As shown in the example of the operation history graph, the lower lip is lowered once with the passage of time (the maximum is reached in 14 ms (about 51 mm)) and then raised. The operation history (operation history graph) measured by the operation measurement unit 14 is output to the data conversion unit 15.

  Returning to FIG. 2, the data conversion unit 15 performs numerical analysis on the operation history (operation history graph) measured by the operation measurement unit 14, thereby converting it into an operation spectrum graph represented by a plurality of preset spectrum components. To convert. The data conversion unit 15 converts each of the four or five operation histories measured by the operation measurement unit 14 into an operation spectrum graph. In this embodiment, Fourier transform is used for numerical analysis. However, in addition to Fourier transform, function approximation that approximates motion history as a function or motion history is analyzed based on the amount of movement of feature points for each minute time. A method or the like may be adopted.

  Thus, the operating spectrum graph has a characteristic peak, and in this example, a peak exists at a frequency of “1” in the detected spectrum. This characteristic peak occurs because the movement of the lip portion is described below when the speaker speaks. That is, the movement of the lip portion is a series of operations that are first opened at the time of utterance, transformed into a certain shape, and closed after a single vowel is pronounced. At this time, in the pronunciation of a single vowel, the motion history graph becomes very periodic, although there are some individual differences in the series of operations from the opening to closing of the lip. As a result, a characteristic frequency always appears at the time of vowel pronunciation.

  Note that when the motion spectrum graph is identified using the Fourier transform in the data conversion means 15, various appearing even if the operation speed (speech speed) when the speaker speaks increases or decreases. The pattern (spectral pattern) is different only in the position of the frequency, and the pattern shape does not change. That is, the data conversion means 15 can obtain an operation spectrum graph corresponding to the utterance content regardless of the utterance speed of the utterer.

  Also, the four feature points at the top, bottom, left, and right ends of the lip part or the five feature points including the lower jaw each move in a unique manner. Therefore, the motion history of the four or five feature points is greatly dependent on the vowel that is pronounced. Will be different. As a result, the motion spectrum graph obtained by converting the motion history graph of these feature points varies greatly depending on the vowel that is pronounced, so that it can be distinguished according to the vowel that is pronounced. For example, in British English, it is said that there are 24 vowels. By analyzing the spectrum of the movement history of lip movements for each of these English vowels (hereinafter also referred to as “English vowels”), It has been confirmed that it can be converted into a motion spectrum graph (Onoe et al .: Lip motion analysis for British English vowels, IPSJ Annual Conference, 2009, 7-3, Onoe et al: Lip motion analysis for British English vowels (Part 2) Video Information Media Society Winter Conference, 2009, 5-9).

  As shown in FIG. 7, in the motion spectrum graph at the left end of the lip, there is a peak at a frequency of “1” in both the vowel string similar to “a” and the vowel string similar to “c”. The vowel string similar to “U” shows a strong operating spectrum, and the vowel string similar to “A” shows a weak operating spectrum. As described above, the motion spectrum graph at the left end of the lip is greatly different between the vowel string similar to “A” and the vowel string similar to “C”. Although not shown here, the operation spectrum graph at the upper lip is different for “look” and “luke” of vowel strings similar to “c”. In this way, the characteristic points from which the characteristic point spectra are obtained differ depending on the vowel, and different operating spectra are obtained.

  Next, with reference to FIG. 8, an example of each operation spectrum graph when a different word is spoken for each English vowel close to “a, i, u, e, o” in Japanese will be described. In FIG. 8, frequency is plotted on the vertical axis, and word pairs including English vowels close to “A” to English vowels close to “A” are arranged in order from the left side on the horizontal axis. For example, a word pair including an English vowel close to “a” is “father” and “cup”. In FIG. 8, “lower lip”, “left lip”, and “lower jaw” shown below each word pair indicate feature points at which a characteristic motion spectrum appears in the word pair. .

Returning to FIG. 2, the difference calculation means 16 analyzes the operation spectrum when the speaker speaks a certain word analyzed by the data conversion means 15 and the exemplary motion spectrum when the instructor speaks the same word. The difference is calculated.
Here, the lip movement is mainly reflected in the first component among the plurality of spectrum components of the movement spectrum. In other words, the first component of the motion spectrum is information indicating the magnitude of the movement of the feature point in the lip motion.

  As shown in FIG. 9A, in the native motion spectrum graph, when the entire motion spectrum is 100%, the ratio of the first component of the motion spectrum to the entire motion spectrum is approximately 25%. . On the other hand, as shown in FIG. 9B, in the motion spectrum graph of a speaker who has just started learning, the ratio of the first component of the motion spectrum to the entire motion spectrum is 100% when the entire motion spectrum is 100%. , Approximately 60 percent. Then, as shown in FIG. 9 (c), the same speaker's motion spectrum graph as in (b) where learning has progressed to some extent, when the entire motion spectrum is 100% among a plurality of motion spectrum components, The ratio of the first component of the operating spectrum to the entire operating spectrum has been reduced to approximately 40 percent.

  According to FIGS. 9A to 9C, even when words containing the same vowel are uttered, there is a difference in the ratio of the first component in the entire motion spectrum due to the difference in the lip motion among the speakers. You can see that it happens. Further, as learning progresses and lip movement is improved, the ratio of the first component of the speaker's motion spectrum to the entire motion spectrum approaches the ratio of the first component of the native motion spectrum to the entire motion spectrum. I understand that.

As described above, the difference between the lip motion of the speaker (speaker) and the lip motion of the instructor is that the first component of each motion spectrum occupies the entire motion spectrum when a word containing the same vowel is spoken. Appears as a difference in ratio.
Since the difference between the ratio of the speaker and the ratio of the leader represents the difference in pronunciation between the speaker and the leader, the correction amount of the lip movement of the speaker is obtained by using this difference. It becomes possible.

  Therefore, in the difference calculation means 16, the instructor uttered the same word from the ratio of the first component of the motion spectrum in the whole motion spectrum when the speaker utters the word specified by the speech content specification means 12. The difference obtained by subtracting the ratio of the first component of the motion spectrum in the entire motion spectrum is calculated for each feature point.

  When the difference calculated by subtracting the ratio of the instructor from the ratio of the speaker is a negative value, the speaker's mouth opening is smaller than the exemplary lip movement of the instructor. It will be too much. Similarly, when the calculated difference is a positive value, the speaker's mouth opening is too large for the exemplary lip movement of the instructor. In this way, the direction in which the lip movement is corrected (larger direction or smaller direction) can also be determined by the sign of the difference value.

In the following, when distinguishing the difference for each feature point, the upper lip difference is ΔU, the lower lip difference is ΔD, the lip left end difference is ΔL, the lip right end difference is ΔR, and the lower jaw difference ΔJ, and when the feature point in the lip center portion depth direction is extracted by the image processing means 13, the difference in the lip center portion depth direction is expressed as ΔDepth.
The difference for each feature point calculated by the difference calculation unit 16 is output to the correction amount calculation unit 17.

Note that the exemplary motion spectrum of the instructor asks the instructor to utter all vowels (or words that contain vowels) in that language in advance, and the lip portion of the instructor at that time The photographing means 20 takes a picture each time, the image processing means 13 extracts a preset position of the feature point from the image acquired by the photographing means 20, and the action measuring means 14 measures the action history of the feature point. Then, the operation spectrum for each vowel can be obtained by numerically analyzing the operation history by the data conversion means 15.
Then, the difference calculating means 16 can calculate the ratio of the first component of the motion spectrum in the entire motion spectrum for each vowel.

  Since the ratio of the instructor is stored in the data storage means 11 in association with the motion spectrum as described above, the difference calculation means 16 is appropriately selected when the motion spectrum of the speaker is input. Read.

The correction amount calculating unit 17 calculates the correction amount of the lip movement of the speaker based on the difference calculated by the difference calculating unit 16.
The correction amount calculating unit 17 compares the absolute value of the difference for each feature point calculated by the difference calculating unit 16 with a threshold value predetermined for each feature point, and determines that the absolute value of the difference is larger than the threshold value. For points, the correction amount is calculated according to a correction function predetermined for each feature point.
On the other hand, regarding the feature point for which the absolute value of the difference is determined to be smaller than the threshold value, it is assumed that the feature point is sufficiently close to the operation of the instructor's feature point, and no modification of the speaker's feature point operation is instructed. This threshold value is determined in advance for each vowel by conducting an experiment or the like.

Here, the correction function applied in the present embodiment will be described with reference to FIG.
FIG. 10 is a graph of a correction function showing the relationship between the upper lip difference ΔU and the upper lip motion correction amount f1, and the vertical axis represents the upper lip motion correction amount f1 and the horizontal axis represents the correction value f1. The difference ΔU is taken. In FIG. 10, f1max indicates the amount of correction of the movement of the speaker's upper lip when the upper lip is moved most greatly when the designated vowel is pronounced, and f1min is the pronunciation of the designated vowel. Sometimes, the correction amount of the motion of the speaker's upper lip when the upper lip is moved the smallest is shown, and f1 (ΔU) is a correction formula that linearly connects these two points. F1max in FIG. 10 is the correction amount when the upper lip is moved most greatly, and varies depending on the person. f1min corresponds to the correction amount when the upper lip is not moved. In carrying out the present invention, it is not necessary to measure these values, but it conceptually indicates that (ΔU) has an upper limit and a lower limit as f1max and f1min. However, f1min includes the case where the feature points of the lip left end, lip right end, upper lip, lower lip, and lower jaw are not necessarily moved but moved in the opposite direction. For example, it is a case where the mouth is shrugged with pronunciation that should widen the mouth sideways.

  In FIG. 10, −Thu is a threshold value for determining whether or not the upper lip is corrected in a direction to be opened widely, and + Thu is a threshold value for determining whether or not the upper lip is corrected in a direction to be opened small. It is. When absΔU that is the absolute value of the difference ΔU is smaller than the threshold value Thu (when it falls within the threshold value + Thu to −Thu), the corresponding correction amount f1 (+ Thu) to f1 (−Thu) is instructed to the speaker as the correction amount. Or the process of substituting ΔU into the correction function is not performed.

  Since the physical size of the lips differs depending on the person and the amount of lip movement varies, when applying the correction function to the speaker, the lip movement amount of the instructor stored in advance in the data storage unit 11 is used as a reference. Normalization is performed on. As an example of the method, the difference between the left lip and the right lip when the mouth is naturally closed, that is, the width and the difference between the upper lip and the lower lip, ie, the vertical width Further, the widths of the instructor and the inventor are compared with each other, and the ratios are obtained. Then, using the ratios obtained in this way, the lip movement amount of the speaker measured during the utterance learning is normalized.

  And when correcting linearly as shown in FIG. 10 according to the correction function f1 (ΔU) normalized by the speaker, when the difference ΔU is larger than a predetermined threshold value + Thu, as the distance from the threshold value + Thu increases, The correction amount of the upper lip movement is increased in the direction of decreasing the upper lip movement of the speaker. On the other hand, when ΔU is smaller than the threshold value -Thu, the correction amount of the upper lip movement is increased in a direction in which the upper lip movement increases as the distance from the threshold -Thu increases. Thus, according to the correction function f1 (ΔU), the direction and magnitude of the correction of the movement of the speaker's upper lip in order to bring the movement of the speaker's upper lip closer to the movement of the instructor's upper lip is specified. The correction amount can be calculated.

  Further, for example, when the correction amount of the movement of the left lip of the lip is obtained, the difference ΔL (not shown) is used when the correction is performed linearly as shown in FIG. 10 according to the correction function of the lip difference end normalized by the speaker. ) Is larger than a predetermined threshold value Thl (not shown), the movement of the left lip portion of the lip is corrected in a direction to reduce the movement of the left lip portion of the speaker as the distance from the threshold value Thl (not shown) increases. Increase the amount. On the other hand, when ΔL (not shown) is smaller than a predetermined threshold value −Thl (not shown), the left edge of the lip increases in a direction in which the movement of the left edge of the lip increases as the distance from the threshold value −Thl (not shown) increases. Increase the amount of movement correction.

  Further, for example, when obtaining the correction amount of the motion in the depth direction of the lip center portion, when correcting linearly as shown in FIG. 10 according to the correction function in the depth direction of the lip center portion normalized to the speaker, the difference When ΔDepth (not shown) is larger than a predetermined threshold Thdepth (not shown), the distance from the threshold Thdepth (not shown) increases in the direction in which the center of the lip of the speaker is reduced. Increase the amount of motion correction in the depth direction. On the other hand, when ΔDepth (not shown) is smaller than a predetermined threshold value -Thdepth (not shown), the depth of the lip center portion protrudes in the direction of protruding the lip center portion as the distance from the threshold value -Thdepth (not shown) increases. Increase the amount of direction movement correction. Similarly, the correction amount can be calculated for other feature points in accordance with the corresponding correction functions. The correction amount calculation unit 17 outputs the calculated correction amount to the correction information output unit 18.

  The correction information output means 18 is a means for outputting the correction amount calculated by the correction amount calculation means 17 as correction information in a format that can be recognized by the speaker. This correction information indicates the correction direction and size of the feature point. Here, a pattern generated in advance according to the correction amount is stored in a storage unit (not shown), and is appropriately read out by the correction information output unit 18. However, the correction information output unit 18 may have a function of generating a pattern each time according to the correction amount calculated by the correction amount calculation unit 17.

  The correction direction is indicated by an upward arrow when the upper lip is corrected to move further upward, and by an downward arrow when the correction is made to reduce the upward movement. Similarly, when the lower lip is corrected so as to move further downward, it is indicated by a downward arrow, and when it is corrected so as to reduce the downward movement, it is indicated by an upward arrow. When the left lip of the lip is corrected to move in the direction in which the mouth is squeezed, it is indicated by a right arrow, and when it is corrected to move in the opening direction, it is indicated by a left arrow. On the contrary, the right end of the lip is indicated by a left arrow when it is corrected so as to move the mouth, and it is indicated by a right arrow when it is corrected so that it moves in the opening direction. When the lower jaw is corrected to move further downward, it is indicated by a downward arrow, and when it is corrected so as to reduce the downward movement, it is indicated by an upward arrow. When correcting the center of the lip so as to protrude further, an arrow that protrudes is shown in a perspective view in the case of displaying in a plane, and in the case of stereoscopic display, it is indicated by an arrow protruding with parallax. When the center of the lip is corrected so as not to protrude too much, it is indicated by an arrow in the reverse direction. The size of the arrow differs depending on the correction amount, and may be determined empirically such as large, medium, small, and large depending on the size of the correction amount.

  This pattern may be in any form that can be grasped by the speaker and can indicate the correction direction and size of the feature point. For example, the pattern may be a sound that can be reproduced by a speaker (not shown) of the display device 30, or may be a text that can be displayed on the display screen 31 of the display device 30.

Further, for example, the pattern is a CG (Computer Graphics) of a graphic (for example, an arrow) indicating the direction and size of correction, and the correction amount is calculated by the correction amount calculation means 17 in the lip portion image of the speaker. A synthesized image synthesized at a position corresponding to the calculated feature point may be used as the correction information. When this figure is an arrow, the direction of the correction can be indicated by the direction of the arrow, and the size of the correction can be indicated by the length of the arrow. For example, the longer the arrow, the larger the correction amount, and the shorter the arrow, the smaller the correction amount. When the correction amount is represented by the length of the arrow, the length of the arrow is set in advance according to the correction amount.
Then, when the CG of the arrow is combined with the image of the lip portion of the speaker, the base of the arrow may be aligned with the position corresponding to the feature point for correcting the motion in the image. This makes it easier for the speaker to intuitively understand which part of the lip should be corrected and how it depends on the direction of the arrow (whether the arrow points inward or outward). Become.

  If the display screen 31 can display an image in three dimensions, the pattern may be a three-dimensional CG. According to this, when the correction amount calculation unit 17 calculates the correction amount of the motion in the depth direction of the lip center, the correction information output unit 18 indicates the correction direction and size in the lip center. Three-dimensional arrow CG can be synthesized to obtain correction information. In this case, the direction of correction can be indicated by whether the tip of the arrow is directed toward the front of the screen or toward the back of the screen.

As described above, when the correction information is a composite image, the correction information output unit 18 receives the input of the correction amount of a certain feature point from the correction amount calculation unit 17 and then the image of the lip portion of the speaker from the photographing unit 20. And a graphic CG corresponding to the correction amount of the feature point is read from a storage unit (not shown), and a composite image is generated by synthesizing the graphic CG of the lip portion of the speaker. When the correction information is a pattern stored in a storage unit (not shown) in advance, the correction information output unit 18 receives an input of a correction amount of a certain feature point from the correction amount calculation unit 17 and then stores the correction information from the storage unit (not shown). A pattern (speech, text, etc.) corresponding to the correction amount of the feature point is read out.
In this way, the correction information generated by the correction information output unit 18 or read from the storage unit (not shown) is output to the display device 30.

  Then, the correction information output from the correction information output means 18 to the display device 30 is displayed on the display screen 31 of the display device 30, or reproduced by a speaker (not shown) of the display device 30, thereby allowing the speaker to speak. It is possible to objectively recognize the improvement of lip movement.

  Here, with reference to FIG. 11, a screen configuration when the correction information is output from the correction information output unit 18 to the display device 30 and the correction information is displayed on the display screen 31 will be described. Here, the utterance learning support device 1 determines that the lip movement at the left end of the speaker's lip is too larger than the movement at the left end of the instructor by the correction amount calculation means 17, and It is assumed that the correction amount for making the operation small is calculated and the correction information output means 18 outputs the correction information corresponding to the correction amount to the display device 30.

  Here, the utterance learning support device 1 uses the correction information output means 18 to display a figure (here, an arrow) indicating the magnitude and direction of correction at a position corresponding to the left end of the lip on the lip portion image of the speaker. Assume that an image obtained by synthesizing CG and text indicating the size and direction of correction are output to the display device 30 as correction information.

  As shown in FIG. 11, the display screen 31 has the lip as a correction information output from the correction information output means 18 of the utterance learning support device 1 at a position corresponding to the left end of the lip in the image of the speaker's lip. An image Gs in which an arrow Y having a predetermined length extending from the left end toward the right end of the lip is combined is displayed. Further, below the image Gs, text data Tb “Please move the left end of the lip a little smaller” is displayed as the correction information output from the correction information output means 18 of the utterance learning support device 1. In this way, the correction information output from the correction information output means 18 of the utterance learning support device 1 is displayed on the display screen 31 of the display device 30 so that the speaker can objectively improve the improvement of his / her lip movement. Can be recognized. As a result of correcting the movement of a certain feature point, the movement of another feature point may deviate from the movement of the instructor. In that case, it is determined whether the difference is within the threshold for the feature point, and if not, the correction amount is calculated and shown to the speaker as correction information, and the speaker who confirmed the correction information speaks again Then, it is determined again whether or not the difference is within the threshold value, and the speaker is finally set so that the difference between the movement of all the feature points and the movement of the corresponding feature point of the instructor falls within the threshold value. Continue training.

  In the utterance learning support device 1 described above, each means can be realized as a function program in a computer, and the function programs can be combined to operate as an utterance learning support program.

[Operation of speech learning support device]
Next, the operation of the utterance learning support system including the utterance learning support device 1 will be described with reference to FIG.
In the utterance learning support system, the utterance content designation unit 12 of the utterance learning support device 1 receives from the display device 30 an input of a signal indicating that the utterer has decided to start learning a certain language (step S11). . Then, the utterance learning support system is included in a plurality of data sets of the language stored in the data storage unit 11 in response to the signal received in step S11 by the utterance content designation unit 12 of the utterance learning support device 1. A word to be uttered by the speaker from among the words to be spoken, text data for instructing the utterance of the word, and an image of the lip portion when the instructor utters the word, and a display device 30 is displayed on the display screen 31 (step S12). At this time, when the voice data of the word is further stored in the data storage unit 11, the utterance content designation unit 12 further reads out the voice data and reproduces it from a speaker (not shown) of the display device 30. May be.

Subsequently, in the utterance learning support system, the photographing unit 20 photographs the lip portion when the speaker utters the word instructed in step S12 (step S13).
Then, when the image processing means 13 of the utterance learning support device 1 receives the input of the lip portion image of the utterer taken in step S13, the utterance learning support system receives the lip movement reference from the lip portion image. The position of the preset feature point is extracted (step S14).

  Then, the speech learning support system measures the motion history of the feature point whose position is extracted in step S14 by the motion measurement means 14 of the speech learning support device 1 (step S15). Then, the utterance learning support device 1 uses the data conversion means 15 to Fourier-analyze the motion history for each feature point measured in step S15 and generate a motion spectrum for each feature point (step S16).

  Furthermore, the utterance learning support system obtains the ratio of the first component of the motion spectrum in the motion spectrum graph for each feature point generated in step S16 by the difference calculation means 16 of the utterance learning support device 1 to the entire motion spectrum, The difference between this ratio and the ratio of the first component of the motion spectrum obtained from the typical motion spectrum graph of the instructor stored in advance in the data storage means 11 to the entire motion spectrum is calculated for each feature point ( Step S17).

Then, in the utterance learning support system, the absolute value of the difference for each feature point calculated in step S17 by the correction amount calculation means 17 of the utterance learning support device 1 is a threshold value determined in advance for the vowel of the word uttered by the speaker. Whether it is larger is determined for each feature point (step S18). When it is determined that there is a feature point whose absolute value of the difference is larger than a predetermined threshold (Yes in step S18), the utterance learning support device 1 uses the correction amount calculation unit 17 to calculate the difference between the feature points. Is substituted into a predetermined correction function, a correction amount specifying the correction direction and magnitude according to the difference is calculated (step S19), and the process directly proceeds to step S20.
On the other hand, when it is determined that there is no feature point whose absolute value of the difference is larger than the predetermined threshold (No in step S18), the utterance learning support device 1 does not calculate the correction amount by the correction amount calculation unit 17. Then, the process is finished as it is.

  Then, the utterance learning support system reads correction information corresponding to the correction amount from the storage unit (not shown) for the feature point whose correction amount is calculated in step S19 by the correction information output unit 18 of the utterance learning support device 1. Or the correction information according to the said correction amount is produced | generated, and it outputs to the display apparatus 30 (step S20).

Then, the utterance learning support system receives the input of the correction information output from the utterance learning support device 1 in step S20 by the display device 30. Then, when the correction information received by the display device 30 is an image or text, the utterance learning support system displays the correction information on the display screen 31 and, in the case of voice, reproduces it from a speaker (not shown). (Step S21).
Then, the utterance learning support system returns to step S13 and refers to the correction information displayed on the display screen 31, for example, and the lip when the speaker repeatedly utters the word already specified by the utterance content specifying means 12 The part is photographed by the photographing means 20. In this way, the utterance learning support system repeats step S13 to step S21 until it is determined in step S18 that there is no feature point whose absolute value of the difference is larger than the predetermined threshold value.
As described above, the utterance learning support system can indicate the correction amount of the lip movement to the speaker in a manner that the speaker can objectively recognize.

According to the utterance learning support device 1 according to the present embodiment, as a result of comparing the lip movement of the speaker with the lip movement of the instructor, the speaker can objectively recognize the correction amount of the lip movement of the speaker. It can be shown in an embodiment. For this reason, it becomes possible to make a speaker objectively recognize the improvement of his / her lip movement. Thereby, it can be used for effective learning education.
Moreover, according to the utterance learning support device 1 according to the present embodiment, it is possible to help a person who is naturally deaf or deaf, such as a narrator, to acquire correct pronunciation when learning speech. .
Furthermore, in the speech learning support device 1 according to the present embodiment, a patient who has troubled the movement of the left part or the right part of the lip due to hemiplegia due to cerebral infarction, stroke or the like correctly pronounces a word. Can be used for rehabilitation of lip movements.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can be appropriately changed without departing from the spirit of the present invention.
For example, in the above-described embodiment, the correction function applied when the correction amount is calculated by the correction amount calculation unit 17 is a linear expression as shown in FIG. 10, but is not limited to this, and the absolute value of the difference greatly deviates from the threshold value. In order to increase the correction amount at the time and reduce the correction amount when the absolute value of the difference is close to the threshold value, a quadratic polynomial may be used. In addition, when priority is given to the delicate correction of the lip movement when the absolute value of the difference is close to the threshold value, a square root (0.5th order equation) or a logarithmic polynomial may be used.

  In the embodiment, the word is stored in the data storage unit 11 in a text format, and the text data of the word is read from the data storage unit 11 by the utterance content specifying unit 12 and displayed on the display screen 31 of the display device 30. However, the present invention is not limited to this, and the data storage means 11 stores voice data obtained by voice-synthesizing the word in association with the word. The voice data may be read by the designation means 12 and reproduced from a speaker (not shown) built in the display device 30 to indicate a word to be spoken to the speaker. Further, both voice and text may be used.

  When a stereo camera is used as the photographing unit 20, for example, a horizontal coordinate difference between the lower lips of the left and right cameras constituting the stereo camera is calculated as a parallax, and a camera such as a known inter-camera distance of the stereo camera is calculated. Applying the parameters, the movement in the depth direction of the lower lip is calculated by the principle of stereo surveying. The movement of the upper end of the lip in the depth direction can be calculated in the same manner.

  Next, an utterance learning support device 1B according to a modification of the embodiment will be described with reference to FIG. The utterance learning support device 1B according to the modification includes an evaluation value calculation means 19 in addition to the configuration of the utterance learning support device 1 according to the embodiment. In the following description, components that are the same as those in the above embodiment are given the same reference numerals and description thereof is omitted.

  As shown in FIG. 13, the utterance learning support device 1B includes a data storage unit 11, an utterance content designation unit 12, an image processing unit 13, an action measurement unit 14, a data conversion unit 15, and a difference calculation unit 16. , Correction amount calculation means 17, correction information output means 18, and evaluation value calculation means 19.

The evaluation value calculation means 19 calculates an evaluation value of the lip movement of the speaker based on the result of comparing the lip movement of the speaker with the lip movement of the instructor.
Here, the evaluation value calculation means 19 calculates the total spectrum by multiplying all the spectrums constituting the motion spectrum graph of the speaker obtained by the data conversion means 15 with respective weights, as shown in the equation (1). Furthermore, the total spectrum is calculated by multiplying all the spectrums constituting the instructor's motion spectrum graph stored in the data storage means 11 with respective weights, and the first spectrum of the motion spectrum is transmitted to the speaker. The amount that the component occupies in the total spectrum is calculated as the evaluation value X.

In the formula (1), T ₁ represents the first component of the operating spectral leaders, P ₁ shows a first component of the operating spectral of a speaker (learner). In Equation (1), the weight W _i can be set as appropriate, but the weight W ₁ applied to the first component of the operating spectrum is set to be larger than the weights of the other components.

By calculating the evaluation value X by the evaluation value calculation means 19 in this way, it is possible to objectively determine whether or not the lip movement of the speaker approaches the lip movement of the instructor due to the change in the evaluation value before and after practice. Can be recognized.
The evaluation value X calculated by the evaluation value calculation means 19 is output to the display device 30 and displayed on the display screen 31. This evaluation value X may be displayed on the display screen 31 together with the correction information output from the correction information output means 18 to the display device 30, or only one of them may be displayed on the display screen 31. Displaying the evaluation value X together with the correction information on the display screen 31 is preferable because it makes it easier for the speaker to recognize the improvement of his / her lip movement.

DESCRIPTION OF SYMBOLS 1, 1B Utterance learning support apparatus 11 Data storage means 12 Utterance content designation means 13 Image processing means 14 Motion measurement means 15 Data conversion means 16 Difference calculation means 17 Correction amount calculation means 18 Correction information output means 19 Evaluation value calculation means 20 Imaging means 30 Display device 31 Display screen

Claims (4)

The lip movement of the speaker is obtained from the image of the lip part when the speaker speaks a predetermined word in a certain language, which is photographed by the photographing means, and the lip movement and the instructor An utterance learning support device showing an improvement point of the lip movement of a speaker based on a result of comparison with an exemplary lip movement when a word is uttered,
Data storage means for storing a plurality of associations between at least the word to be uttered by a speaker and an image of a lip portion when the instructor utters the word;
Of the plurality of words stored in the data storage means, a word to be spoken by a speaker is designated by an external input or in a preset order, and the utterance of the word is designated by the speaker. Utterance content designation means for instructing
An image for extracting a plurality of positions of preset feature points that serve as a reference for specifying the lip movement of the speaker from the image of the lip portion when the speaker utters the word specified by the utterance content specifying means Processing means;
An action measuring means for measuring a change in position of each feature point extracted by the image processing means as an action history that is a history of lip action;
Data conversion means for converting the motion history for each feature point measured by the motion measurement means into a motion spectrum represented by a plurality of preset spectral components for each feature point;
Difference calculating means for calculating a difference between the motion spectrum for each feature point determined by the data conversion means and the exemplary motion spectrum for each feature point determined in advance from the image of the exemplary lip portion When,
The absolute value of the difference calculated by the difference calculation means and a predetermined threshold value are compared for each feature point, and when there is the feature point where the absolute value of the difference is larger than the predetermined threshold value, A correction amount calculating means for calculating a correction amount specifying a direction and a size for correcting the operation of the feature point by a predetermined correction function;
Speech learning support device characterized by and a correction information output means for outputting the corrected information in accordance with the correction amount calculated in Viewing device by said correction calculation means. The correction information output means includes
On the image of the lip portion of the speaker, an image indicating the direction and size for correcting the operation of the feature point is synthesized at a position corresponding to the feature point for which the correction amount is calculated by the correction amount calculation unit. The utterance learning support device according to claim 1, wherein the utterance learning support device is output to the display device. The correction information output means includes
2. The text specifying the feature point for which the correction amount has been calculated by the correction amount calculating means and the direction and magnitude for correcting the operation of the feature point are output to the display device. The speech learning support device according to Item 2. The lip movement of the speaker is obtained from the image of the lip part when the speaker speaks a predetermined word in a certain language, which is photographed by the photographing means, and the lip movement and the instructor In order to show the improvement of the lip movement of the speaker based on the result of comparing the exemplary lip movement when speaking a word, the computer,
Wherein the word to be uttered at least the speaker, from among the plurality of words stored in data storage means for storing a plurality of correspondence between images of the lip portion when the leader has uttered the word, calling speaker Utterance content designation means for designating one word to be uttered from outside or one in a preset order, and instructing the utterer to utter the word,
An image for extracting a plurality of positions of preset feature points that serve as a reference for specifying the lip movement of the speaker from the image of the lip portion when the speaker utters the word specified by the utterance content specifying means Processing means,
An action measuring means for measuring a change in position for each feature point extracted by the image processing means as an action history that is a history of lip movement;
A data conversion means for converting the motion history for each feature point measured by the motion measurement means into a motion spectrum represented by a plurality of preset spectral components for each feature point;
Difference calculating means for calculating a difference between the motion spectrum for each feature point determined by the data conversion means and the exemplary motion spectrum for each feature point determined in advance from the image of the exemplary lip portion ,
The absolute value of the difference calculated by the difference calculation means and a predetermined threshold value are compared for each feature point, and when there is the feature point where the absolute value of the difference is larger than the predetermined threshold value, A correction amount calculating means for calculating a correction amount specifying a direction and a size for correcting the operation of the feature point by a predetermined correction function;
Speech learning support program for causing a correction information corresponding to the correction amount calculated by the correction amount calculating unit as, correction information output means for outputting the Viewing device.

Images