JPH1097280A - Speech image recognition and translation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the precision of translation by judging a phrase of a sentence consisting of an inputted continuous speech when features of an image of the whole mouth and its periphery corresponding to features of a speech indicate the end of the phrase of the sentence. SOLUTION: The device consists of a microphone 101, A/D converting ICs 102 and 107, a speech input part 103, a memories 104 and 109, a speech recognizing process part 105, a camera 106, an image input part 108, an image recognizing process part 110, a translating process part 111, a speech synthesizing process part 112, a D/A converting IC 113, an LCD 114, and a speaker 115. A phrase of a sentence consisting of a continuous input speech is recognized and the words constituting the sentence are recognized to perform speech recognition and translation. Here, when features of the image of the mouth corresponding to the speech indicate the end of the phrase of the sentence, it is judged that it is the phrase of the sentence consisting of the inputted continuous speech, the character and character string in the ending of this phrase are recognized, and a mark indicating the phrase is added to accurately recognize the phrase and word ending of the speech input sentence.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a speech image recognition translator suitable for application to a portable information translator such as a portable speech translator or a PDA, a portable translator, and the like. The present invention relates to a speech recognition translator for bidirectionally taking in necessary conversational speech, recognizing the speech, and translating the speech into each other's native language.

[0002]

2. Description of the Related Art With the rapid increase in the number of overseas travelers, portable translators such as a fixed-volume collection of voices are becoming widespread in order to overcome the inconvenience of communication due to language barriers. These portable translators employ a system in which sentences used for conversation are registered in advance as voice data, and necessary conversation sentences are selected and reproduced according to the situation at the time. Therefore, in this method, one simply asks one's own question or request in the language (language) of the other party by selecting and expressing from a fixed range of fixed sentences, and using his / her own language with some freedom I can't translate words or other people's words.

Japanese Patent Application Laid-Open No. 5-35776 (name: "Translator with automatic language selection function") recognizes and translates the voice of an operator input from a microphone, and outputs the voice of the translated language. A portable translation device adapted to do so is disclosed.

FIG. 10 is a block diagram showing an example of such a conventional speech translator. In FIG.
1 is a control unit, 1002 is a speech section cutout unit, 1003
Is a voice recognition unit, 1004 is a display unit, 1005 is a voice synthesis unit, 1006 is a memory card for translated word data, 1007
Denotes a voice recognition dictionary unit, 1008 denotes a speaker, 1009 denotes a microphone, 1010 denotes a speaker amplifier, and 1011 denotes an operation signal.

[0005] The control unit 1001 is mainly composed of a microprocessor, and controls each unit of the apparatus.

[0006] The voice section extracting unit 1002 is provided with a microphone 1
The voice input from 009 is converted into a digital signal, cut out, and sent to the voice recognition unit 1003.

The voice recognition unit 1003 analyzes the voice input from the microphone 1009 and cut out through the voice section cutout unit 1002 according to the instruction of the control unit 1001 which receives the operation signal 1011 input from the keyboard or the switch. Then, the result is input to the speech recognition dictionary unit 10.
The voice recognition is performed in comparison with the standard voice pattern stored in 07.

[0008] The speech synthesizing unit 1005 includes a speech recognizing unit 100.
3 is read from the translated word data memory card 1006, converted into a voice signal, and output as a voice via the speaker amplifier 1010 and the speaker 1008.

[0009] The display unit 1004 displays instructions of the user of the translation apparatus, displays translated characters, and the like.

[0010] The translated word data memory card 1006 is
It is composed of a ROM, a flash memory, a hard disk, etc., and stores speech data when a translated word is synthesized and output. From the translated word data memory card 1006, a character code corresponding to the translated word is read and displayed on the display unit 1004. By exchanging the translated word data memory card 1006 with another language, it becomes possible to translate the data into a plurality of languages.

The speech recognition dictionary unit 1007 has a ROM, RA
And a standard voice pattern corresponding to the utterance of the operator. The standard voice pattern is registered and stored by the operator in advance.

A technique for combining image information in recognizing voice information is disclosed in, for example,
Japanese Patent Application Laid-Open No. 188998 and Japanese Patent Application Laid-Open No. 63-191198 do not disclose a specific recognition method that enables accurate translation.

[0013]

In the above-mentioned conventional portable speech translating apparatus, although the speech generated by the operator is recognized, the translation is performed in advance in the same manner as a translator in the form of a fixed sentence such as a set of conversations by speech. This is functionally the same as a method in which sentences used for conversation are registered as voice data, and necessary conversation sentences are selected and reproduced according to the situation at the place.
In other words, it is possible to ask one's question or request one-sidedly in the other party's language, but since it is not possible to recognize and translate the speech of the unspecified other person's natural conversation, There is a problem that a situation occurs in which a conversation cannot be established because the user cannot understand that the user is in the office. In a portable speech recognition device, it is more important to have a person translate what he / she is saying than to translate what he / she wants to say. There are many difficult technical issues that need to be solved to achieve that.

In order to recognize and translate a continuous voice composed of relatively short sentences, such as a conversation exchanged at an overseas travel destination or the like, words and words constituting the continuous voice, for example, in the case of Japanese Is used to distinguish between nouns, verbs, particles, etc.
Without a clear recognition of sentence breaks, correct translation cannot be performed. Therefore, it is conceivable to input the voice in a unit of a phrase, but an unnatural conversational sentence will be generated even if the input is too long. That is, the speech must be recognized and translated by inputting a continuous speech composed of relatively short sentences, such as a conversation exchanged at an overseas travel destination or the like, which is somewhat conscious of a phrase. .

SUMMARY OF THE INVENTION An object of the present invention is to realize a portable speech recognition translator capable of recognizing and translating mutual conversational speech which is a little like conversation. Another object of the present invention is to provide a speech recognition and translation device capable of recognizing conversational speech by recognizing words constituting a sentence and performing correct translation.

[0016]

In order to achieve the above-mentioned object, a speech recognition / translation apparatus according to the present invention comprises: means for capturing voice and image; a memory for storing captured voice data; A voice recognition processing unit that extracts a feature of the voice from a series of voices and performs a voice recognition process; a memory that stores the captured image data; An image recognition processing unit that extracts and performs image recognition processing, and a translation processing unit that translates the recognition result of words and sentences that have been subjected to speech recognition and image recognition into words or sentences to be translated, and changes over time Speech recognition by recognizing a phrase of a sentence composed of a series of input speeches and recognizing words constituting the sentence from two correlations between features of the speech and features of an image corresponding to the features of the speech. And translate In the speech image recognition and translation device, when the feature of the mouth image corresponding to the voice indicates the end of a sentence phrase, it is determined that the sentence is composed of a series of input speeches, and the end of the phrase is determined. Characters and character strings are recognized, and a mark indicating a phrase is added. In this way, the phrase and the ending (particles in Japanese) of the voice input sentence can be accurately recognized, so that the translation accuracy is improved.

More specifically, a process in which a series of voices captured at the same sampling frequency at the same time and voice characteristics extracted from an image changes from a consonant to a vowel and further from a vowel to a silent state. In the case where the features of the image of the whole mouth and the vicinity of the mouth corresponding to the features of the voice indicate the end of the phrase of the sentence, it is determined that the phrase is a sentence composed of a series of input voices.

In a process in which the features of a series of Japanese voices captured at the same time at the same sampling frequency and voices extracted from images change from consonants to vowels and further from vowels to silence. If the features of the image corresponding to the whole mouth and its surroundings corresponding to the features of the speech indicate the end of a sentence, the sentence is determined to be a sentence consisting of a series of input speeches, and the end of the sentence is determined. To recognize the particles of

Further, the memory for storing the fetched audio data is constituted by a memory (1) which always fetches audio data, and a memory (2) which fetches the audio data when voice is inputted. The memory for storing the captured image data comprises a memory (3) for constantly capturing image data, and a memory (4) for capturing image data when audio is input. When it is determined, the capture of the audio data and the image data is started. When it is determined that the input of the voice is completed, the capture of the audio data and the image data is stopped, and the voice recognition of the stored voice data and the image data is performed. And to translate.

In the process of starting the capture of voice data and image data when it is determined that voice has been input, and the process of reading out the stored voice data and image data, an arbitrarily set period (where a feature of voice appears) (Sampling frequency), the sound data stored in the memory (1) that always fetches the sound data, the intensity Pi of the sound data fetched at the time Ti, and the time T of the immediately preceding cycle.
The value of the difference ΔPi from the intensity Pi-1 of the sound data taken into i-1 is the boundary value Pth of the intensity of the sound data arbitrarily set.
Is exceeded, the sound data is determined to be sound data, and the sound data is written in the memory (2) in order from the next sound data. The writing of the image data into the memory (3) which always takes in the image data from the memory (3) is finished, the next image data is written into the memory (4) in order, and the audio data stored in the memory (1) and the memory (2)
Along with the audio data stored in the memory (3) to form audio waveform data by sequentially reading the data in the direction of the time axis t, and the image data stored in the memory (3) and the image data stored in the memory (4) together with the time axis t The image data is read out sequentially in the direction to form a series of mouth movement image data.

Further, there is further provided a recognition result correcting unit for correcting or correcting the recognition result of the words and sentences recognized by the voice recognition and the image recognition.

The features of the image are extracted from the face image of the person, the whole mouth, and the image in the vicinity of the periphery, which are normalized by performing resolution conversion and binarization processing of the image data.

Also, the means for capturing the sound and the image may include:
A camera and a microphone are integrally configured to constitute a portable voice image recognition and translation device.

By using the speech recognition and translating apparatus having the above-described configuration, a portable speech recognition translator capable of performing mutual speech recognition and translation, which is slightly conversational, is obtained.

[0025]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of a speech image recognition and translation apparatus according to one embodiment of the present invention. The speech image recognition translator shown in FIG. 1 is a portable speech recognition translator. Such a device can be constituted by several LSIs such as a CPU, a memory, and a dedicated IC. In addition, a chip can be formed on a semiconductor element.

In FIG. 1, reference numeral 101 denotes a directional microphone for capturing voice, and captures a conversation voice exchanged at, for example, an airport, a station yard, an airplane, a hotel, a sightseeing spot, a restaurant or shopping, etc. at an overseas travel destination. .

Reference numeral 102 denotes a 16-bit analog / digital (A / D) conversion IC, which is a continuous analog of digital data in which noise outside the audio band is removed by a filter or an amplifier in the microphone 101 and noise processing is performed. The signal is sampled at a sampling frequency of audio, for example, 12 kHz, and is converted into a digital signal.

Reference numeral 103 denotes an audio capturing unit which performs serial / parallel conversion from serial data to parallel data on 16-bit audio data sampled by the A / D conversion IC 102 and temporarily stores the converted data in a register or the like. It is for.

Reference numeral 104 denotes a memory for storing voice data fetched by the voice fetching unit 103, for example, continuous voice data for one phrase of conversation voice, and is necessary only for writing continuous voice data. It is a memory with a minimum capacity. Writing of the continuous audio data to the memory may be performed by software processing such as a CPU or by dedicated hardware.

Reference numeral 105 denotes a voice recognition processing unit,
A series of speech recognition processes, such as digital filtering, speech analysis, speech section detection, collation, and determination, are performed on the continuous speech data written in 04. Here, acoustic model data, dictionary data, and grammar data necessary for speech recognition are registered and stored in a memory or the like in the speech recognition processing unit 105. The voice recognition processing may be performed by software processing such as a CPU and a DSP, or may be performed by dedicated hardware.

Reference numeral 106 denotes a high-resolution camera for capturing an image. This is, for example, a CCD camera,
The movement of the mouth of the person generating the sound is captured as image data in accordance with the conversation sound exchanged at the airport, station yard, airplane, hotel, sightseeing spot, restaurant, shopping, etc. at the overseas travel destination.

Reference numeral 107 denotes a 16-bit analog / digital (A / D) conversion IC, which samples an analog signal from the CCD camera 106 at a frequency of, for example, 12 kHz in synchronization with a sampling frequency of an audio signal into a digital signal. Convert.

Reference numeral 108 denotes an image capturing unit,
This is for performing serial / parallel conversion from serial data to parallel data on 16-bit image data sampled by the D conversion IC 107 and temporarily storing the data in a register or the like.

Reference numeral 109 denotes a memory for storing image data captured by the image capturing unit 108, for example, continuous image data for one phrase of conversation voice.
Further, the memory has a minimum necessary capacity for writing continuous image data. Writing of the continuous image data to the memory may be performed by software processing such as a CPU or by dedicated hardware.

Reference numeral 110 denotes an image recognition processing unit which performs a series of operations such as digital filtering, image conversion, binarization processing, image analysis, feature extraction, collation, and determination on the continuous image data written in the memory 109. Perform image recognition processing. Here, image model data, dictionary data,
The grammar data is registered and stored in a memory or the like in the image recognition processing unit 105. Image recognition processing is performed by CPU
Or by software processing such as a DSP, or by dedicated hardware. Here, the result of the image recognition processing is passed to the voice recognition processing unit 105.

Reference numeral 111 denotes a translation processing unit for performing a translation process on the speech speech recognition result output from the speech recognition processing unit 105 into a language to be translated. Voice recognition processing unit 1
The recognition result output from 05 is, for example, a text sentence of kana-kanji characters such as nouns, particles, verbs, and adverbs in Japanese. In the translation process, the Kana-Kanji mixed sentence is subjected to syntax analysis and sentence generation from a dictionary, grammar rules, examples, and the like, and a translation result is output.

Reference numeral 112 denotes a speech synthesis processing unit that converts the translation result output from the translation processing unit 111 into speech suitable for a conversation sentence and outputs the speech. This speech synthesis processing unit 1
In step 12, in order to make the conversational speech more natural, the pronunciation and accent of the words making up the sentence and the inflection of the entire sentence are optimized, and the speech of the conversational sentence is synthesized, and the other party is heard. It also performs processing to output easy natural sound.

Reference numeral 113 denotes a 16-bit digital / analog (D / A) conversion IC.
The digital signal of the audio output from is transmitted through, for example, a low-pass filter (LPF) to an audio frequency band 12k.
Convert to analog signal at Hz.

Reference numeral 114 denotes a liquid crystal display (L) for displaying the progress of the speech recognition result and the translation result as text.
CD).

Reference numeral 115 denotes a speech recognition result, its progress,
This is a speaker for synthesizing the translation result and outputting the speech.

FIG. 2 illustrates an example of a method of capturing an image of a mouth movement of a person generating a voice, performing image recognition, and decoding the voice content together with the voice recognition as an approach for supporting voice recognition. FIG.

The camera 106 shown in FIG.
The D camera captures an image of the mouth movement of a person who generates sound in synchronization with the sound input.

Reference numeral 201 denotes a still image of a person generating conversational speech at an arbitrary point in time, which is captured in synchronization with the voice input. The resolution of the image is m dots vertically × n dots horizontally × l bit deep. This image 201 is synchronized with the audio input, for example, when the sampling frequency of the audio is set to 12 kHz and the frame for extracting the features of the audio is set to 240 sampling points of the audio,
The data is taken in synchronization with the audio input in units of 20 ms. If a frame unit of 20 ms in audio capture is applied to image capture, 1 ms in 20 ms
Since the image of the frame will be captured, 5 times a second
That is, an image of 0 frame is processed. This means that the resolution on the time axis is higher than 30 frames / sec, which is a current moving image such as NTSC.

A 202 performs a shading process for converting the captured image 201 from a color image to a shading image, and then performs a binarizing process for obtaining a binarized image, thereby extracting a mouth shape feature. This is an image taken out for the purpose.

Reference numeral 203 denotes normalization,
This is an image obtained by performing edge detection and smoothing processing and extracting features such as a contour of a mouth. The resolution of the image 203 is set to the minimum necessary resolution capable of sufficiently representing the features of the image.

FIG. 3 is a diagram for explaining a configuration of a microphone having a built-in camera for capturing voices and images. In the speech image recognition / translation apparatus shown in FIG. 1, a microphone 101 and a camera 106 are integrated. In particular, in a device such as a portable speech recognition translator, it is important to reduce the number of components, reduce costs, and reduce costs. Also, as described with reference to FIG. 2, since image recognition of mouth movement is performed, if a microphone is incorporated in a microphone that captures sound, an image of the mouth that generates sound toward the microphone can be relatively generated. Can be easily captured.

Reference numeral 301 denotes a microphone body having a built-in camera for capturing audio and images. The outer shape may be cylindrical or square. Reference numeral 302 denotes a microphone, which includes a condenser microphone, a resistance microphone, and the like. Reference numeral 303 denotes a lens, which is tuned so as to capture an image of the face or mouth of a person generating sound. 3
Reference numeral 04 denotes a CCD camera, which captures an image of a face or a mouth of a person who generates sound coming from the lens 303.

Reference numeral 305 denotes a cable for transmitting audio and image data, and includes an audio signal cable 306 and an image signal cable 307. In the audio-visual recognition / translation apparatus shown in FIG.
The image signal cable 307 is connected to the conversion IC 102
/ D conversion IC 107 is connected.

FIG. 4 is a diagram for explaining a process in which a conversational sentence based on conversational speech is actually recognized from speech recognition and image recognition and translated in the speech image recognition translation apparatus according to the present invention. FIG.

The content of the example of the conversation sentence based on the input conversation voice is “Kouenha Dokodeska”. The speed at which the input voice is generated is the natural speed of the conversation, and is carefully and clearly uttered to increase the recognition rate.

Reference numeral 401 denotes a speech waveform of “Kouenha Dokodeska” in a conversation sentence example based on the inputted conversation voice. The amplitude of the sound waveform on the time axis t indicates the sound intensity. If the audio sampling frequency is set to 12 kHz for this audio waveform and the number of audio sampling points is 240 for frames for extracting audio features, the audio will be captured in frame units of 20 ms. .

Reference numerals 402 to 404 denote voice feature frames in 20 ms units when the number of voice sampling points is 240 for frames for extracting voice features. In general, the features of speech are features extracted by speech analysis employed in speech recognition. Here, the feature represented by the audio feature frame 402 is “Kouenha”
Is a feature of dynamic speech. Also, voice feature frame 4
The feature represented by 03 is a feature of the static voice of “c”. Again, the feature represented by the audio feature frame 404 is a dynamic audio feature of "docodesca".

Although it is possible to perform speech recognition from such speech feature extraction, erroneous recognition tends to occur at the beginning (at the beginning) or at the end (at the end) of the speech. For example,
"Kouenha" becomes "Kouenhe", and "Dokodeska" becomes "Kokodeska". Also, considering that this speech recognition translation apparatus is used in overseas travel destinations, the recognition rate is further reduced due to ambient noise and the like.
On the other hand, in translation, the translation accuracy changes depending on the result of speech recognition. In particular, if you take the example of translating from Japanese to English, misrecognition occurs at the beginning and end of words,
If "Kouenha" becomes "Kouenhe" or "Dokodeska" becomes "Kokodeska", it cannot be translated correctly. In addition, correct translation becomes difficult if phrases cannot be recognized. Therefore, for translation, we focused on the movement of the mouth at the beginning (start of word) and end (end of word) of speech as a means of improving the accuracy of speech recognition. 405-407
Captures images of mouth movements in correspondence with audio feature frames in 20 ms units when the number of audio sampling points is 240 points, for extracting audio characteristics indicated by audio characteristic frames 402 to 404. 3 is an image in which features are extracted by the processing described with reference to FIG. Here, the features represented by the image 405 are as follows:
It is a feature of the image by the continuous movement of the dynamic mouth shape of "Kouenha". The features represented by the image 406 are as follows:
"C" is a feature of the static mouth-shaped image. again,
The feature represented by the image 407 is a feature of the image due to the continuous movement of the dynamic mouth shape of “docosca”. Therefore, in contrast to the recognition of continuous speech such as conversational sentences, image recognition of mouth movements is performed to improve the recognition accuracy of sentences at the beginning and end of speech, and translation is performed by performing phrase recognition. Accuracy can be increased. In the process of extracting the features of a series of voices and images captured at the same sampling frequency at the same time, the features of the extracted voices change from consonants to vowels, and further from vowels to silence. If the features of the image of the whole mouth and its surroundings corresponding to the features of the above indicate the end of a sentence phrase, it is determined that the sentence is a sentence composed of a series of input speech, Recognize character strings. In the case of Japanese, it is preferable to recognize the particle at the end of a phrase. Furthermore, for example, image recognition is also performed based on the image features due to the continuous movement of the dynamic mouth shape of “Koenha”, and “Park is” is recognized together with voice recognition. Alternatively, even when voice recognition is not performed, “park” is recognized only from image recognition. Thereby, improvement in robustness can be expected.

Reference numerals 408 to 412 denote speech recognition results of a conversation sentence example “Kouenha Codesca” based on the inputted conversation speech. The result is output as a text sentence in Kana-Kanji. Here, by recognizing the words and phrases constituting the sentence, a slash 409 is inserted between “park” 408 and “ha” 410, and between “ha” 410 and “where is” 412. Insert slash 411 into As a result, the translation process can be facilitated and the translation accuracy can be improved.

Reference numeral 413 denotes a translation result of the recognition result “Park / Ha / Where?” And “Whereis the pa”
rk? Is output as text. Then, the translation result 413 is synthesized and output as voice.

FIG. 5 is a diagram for explaining a method of capturing voice and capturing images in the voice image recognition / translation apparatus according to the present invention.

FIG. 5A shows a waveform of the sound of "Yamada" actually generated as a sound by the inventor. FIG. 5 (b)
Is a memory for taking in audio and images.

In FIG. 5A, the horizontal axis (right direction in the figure) represents time t, and the vertical axis (vertical direction in the figure) represents the amplitude intensity of the speech waveform.

The waveform shown in the section 501 is a voice waveform in a state where the voice is always captured. here,
The waveform shown in the zone 1 is a waveform in the silent state, and the waveform shown in the zone 2 is the first audio waveform in which the voice starts from the silent state. When no sound is generated, a waveform similar to that of the area 1 appears in the area 2 since the waveform shown in the area 1 is continuous. Therefore, for example, in a continuously observed sound waveform, the sound waveform data is recorded in the memory only for a certain time region, the new sound data is always stored in the memory, and the old sound data is deleted from the memory. Consider a memory with the minimum required memory capacity. For example, a capacity capable of storing about 0.1 second of audio data is provided.

The memory (1) 506 in FIG.
Is the minimum required to record the sound waveform data in the memory for a certain time region in the continuously observed sound waveform, always store the new sound data in the memory, and erase from the old sound data. This is a memory having a limited memory capacity n. 507 is a write address WA in the memory (1) 506. 508 is a read address RA in the memory (1) 506. When the audio is captured, the audio data is written to the address indicated by the write address WA507, and the write address WA507 is incremented. This operation is repeated in order from the start address of the memory (1) 506, and when writing to the end of the address is completed, the process returns to the start address and repeats the same processing. For example, in the section 501 of FIG. 5A, the waveform data of the silent section 1 is written to the area 1 of the memory (1) 506 in FIG. FIG.
In the section 501 of FIG. 5A, the first voice waveform data of the area 2 where the voice starts from the silent state is written to the area 2 of the memory (1) in FIG. In the section 501, the data of the area 2 is shown before the data of the area 1 because, in the memory (1) 506, the area 2
Is newer than the data in the area 1. Therefore, in order to perform voice recognition, the section 50 is required.
As in the waveform data shown in FIG.
Will be read out before the data of. This allows
Sound data can always be captured. The sound data read from the memory (1) 506 will be described later.

Next, when the first voice waveform at which voice starts from the silent state is observed, it is determined that the voice data to be fetched next is voice, and it is considered that the voice data is stored in another memory. Section 503 in FIG. 5A is audio data. Thereby, the memory capacity can be significantly reduced.

The memory (2) 509 in FIG.
Is a memory for storing the voice data when the first voice waveform at which the voice starts from the silent state is determined to be voice data to be fetched from the next time. 510 is a write address WA in the memory (2) 509. 511 is a read address RA in the memory (2) 509. For example, in the case of recognition of about a word, the memory capacity is set to have a capacity capable of storing voice data for 3 to 5 seconds. In the area 3 in the memory (2), the audio data 3 in the section 503 in FIG. 5A is stored. Here, if it is determined that the audio data is in a silence state again from the audio state and is silence, the writing of the audio data to the memory (2) 509 may be stopped. By the above-described audio capturing process, a series of audio waveform data 1, 2, and 3 in the sections 502 and 503 shown in FIG. 5A are stored in the memory (1) 506 and the memory (2) shown in FIG. ) 509 and the audio waveform data written in these memories (1) and (2) can be read out in time series as shown at 512.

The memory (3) 513 in FIG.
, Like the memory (1) 506, stores image data of mouth movements in a memory for a certain time period in association with the waveform of a sound that is continuously observed, and always stores new image data in the memory. This is a memory having a minimum required memory capacity m for storing and erasing old image data. 5
14 is a write address W in the memory (3) 513
A. 515 is a read address RA in the memory (3) 513. When the image is captured, the image data is written to the address indicated by the write address WA514, and the write address WA514 is incremented. This operation is repeated in order from the start address of the memory (3) 513, and when writing to the end of the address is completed, the operation returns to the start address again and the same processing is repeated. For example, in the section 501 of FIG. 5A, the image data of the mouth movement corresponding to the waveform data of the silent section 1 is written to the area 1 of the memory (3) 513 in FIG. 5B.

Also, in the section 501 of FIG.
The image data of the mouth movement corresponding to the first audio waveform data in the area 2 where the audio starts from the silent state is written in the area 2 of the memory (3) 513 in FIG.
The reason that the data of the area 2 is shown before the data of the area 1 in the section 501 is to indicate that the data of the area 2 is newer than the data of the area 1 in the memory (3) 513. Therefore, in order to perform image recognition corresponding to voice recognition, even in the image, the section 502
In the memory (3) 513, the data in the area 1 is read out before the data in the area 2 as in the waveform data shown in FIG. This makes it possible to always capture image data corresponding to sound data.

Next, if the first voice waveform at which the voice starts from the silent state is observed, it is determined that the voice data to be taken in next is voice, and the image data is similarly stored in another memory. Think about doing. A section 503 in FIG. 5A is audio data, and there is image data representing a dynamic mouth movement corresponding to the audio data.
Thereby, the memory capacity can be significantly reduced.

The memory (4) 516 in FIG.
If the first sound waveform at which sound starts from the silent state is observed, it is determined that the sound data taken in next is sound, and the image data corresponding to the sound data is similarly determined in the image data. It is a memory for storing. 517 is a write address WA in the memory (4) 516. 518 is a read address RA in the memory (4) 516. This memory (4) 516
5 stores image data corresponding to the audio data 3 in the section 503 in FIG. 5A. Here, the audio data is changed from the audio state to the silence state again,
If it is determined that there is no sound, writing of image data to the memory (4) 516 may be stopped.

By the above described image fetching process, the image data corresponding to the series of audio waveform data 1, 2, and 3 in the sections 502 and 503 shown in FIG.
The memory (3) 513 and the memory (4) 51 shown in FIG.
6 and written in these memories (3) and (4) can be read out in time series as shown at 519.

FIG. 6 is a flowchart showing the voice and image capturing process in FIG.

In processing step 601, sound data is always taken and written to the memory (1), and image data is always taken and written to the memory (3). For example, the sound data is sound data including sound sampled at 12 kHz. The image data is face image data of a person including mouth movements sampled in synchronization with the capture of audio.

In processing step 602, the amplitude intensity P of the sound data is written to the sound data written in the memory (1).
Is observed for each sampled data, and its amplitude intensity Pi is arbitrarily set to a threshold level Pt.
h, the write address WA507,
514 is incremented, and the process returns to repeat the processing of the processing step 601. Also, this processing step 602
With respect to the sound data written in the memory (1), the amplitude intensity P of the sound data is observed for each sampled data, and when the amplitude intensity Pi exceeds an arbitrarily set threshold level Pth, It is determined that the sound data is voice.

In processing step 603, processing step 6
When it is determined in step 02 that the sound data is sound, the write addresses WA507 and 514 of the memory (1) 506 and the memory (3) 513 are stored.

In the processing step 604, it is determined that the sound data to be sampled next is sound data, the sound data Pi + 1 is written to the memory (2) 509, and the image data Ii is written to the memory (4) 516. Write from +1.
As described above, voice data and image data necessary for voice recognition can be captured.

At processing step 605, the voice data written in the memory (1) 506 and the memory (2) 509 and the voice data written in the memory (3) 513 and the memory (4) 516 for voice recognition. In reading the image data, the next address of the write address WA507 of the memory (1) 506 stored in the processing step 603 is set as the head address RA (= WA + 1) 508 of the read address of the memory (1) 506, (1) Read out all audio data stored in 506. Further, the next address of the write address WA514 of the memory (3) 513 stored in the processing step 603 is set as the head address RA (= WA + 1) 515 of the read address of the memory (3) 513, and the memory (3) 5
13 is read out.

Finally, in the processing step 606, the head address RA (head) 511 of the read address is stored in the memory (2) 509 storing most of the audio data.
From the memory (2) 509. In addition, the memory 4 (516) storing most of the image data is read from the head address RA (head) 518 of the read address to the memory (4) 51.
6 is read out.

FIG. 7 shows an example of a method for always taking in sound data and determining whether the data is in a silent state or a sound. In FIG. 5A, a waveform 504 is a waveform that is first taken in when it is determined to be a voice, and 505 is a trigger (flag) or a signal generated when it is determined that the voice is a voice.
A closer look at the first captured waveform determined to be this voice has a waveform as shown by waveform 701 in FIG. Here, reference numeral 702 denotes audio data Pi sampled at time t (i). 703 is audio data P sampled at time t (i-1).
i-1.

For example, the process 602 in FIG. 6 may be performed by observing the difference value ΔP between the audio data Pi 702 and the audio data Pi-1 703 and comparing it with an arbitrarily set threshold level Pth. Further, the integrated value ΣΔP of the difference value ΔP is observed and compared with an arbitrarily set threshold level Pth.
May be performed. Further, by observing the difference value ΔP a plurality of times (k times) within a certain time, the number of observations may be compared with an arbitrarily set number of times (threshold number), and the process 602 in FIG. 6 may be performed. Good.

FIG. 8 is a diagram showing an example of an image and an appearance of the above-mentioned portable audio-visual translator. FIG. 8 (a)
1 shows a situation where an overseas traveler uses the portable speech translator according to the present invention. Overseas travelers who are users, for example, when talking with a clerk at shopping via the display and voice input / output means of the portable translator, translate what they speak into words that the other party can understand, and express their intentions. To convey, and conversely, translate the words of the other person into words that you understand, and understand the intention of the other person. In particular, Japanese, English, German, French, Italian,
Languages such as Russian and Chinese can correspond to the languages of each country and are not limited. FIG.
(B) is an external view of a portable translator, and 801 is a main body of the portable translator.

Reference numeral 802 denotes a multidirectional microphone, which reduces noise at various places included in conversational voices in airports, train stations, airplanes, vehicles such as buses, subways and taxis, and in tourist attractions. It is used for the purpose of removing, and when there is no conversation voice, the whole sound at each location is captured.

Reference numeral 803 denotes a microphone having a built-in CCD camera as shown in FIG. 3, which has directivity, and is a conversation voice exchanged at an airport, a station yard, an airplane, a hotel, a sightseeing spot, a restaurant or shopping, etc. at an overseas destination. And an image of the face including the movement of the mouth is captured as an analog signal.

Reference numeral 804 denotes a voice output means, which comprises a speaker or an earphone for reporting the content translated by voice recognition.

Reference numeral 805 denotes the result of speech recognition and its modification,
10 is a display for displaying the contents of a correction result and a translation result.

An IC card 806 stores, for example, an acoustic model for speech recognition translation from Japanese into Chinese, a word dictionary, a grammar dictionary, a translation example dictionary, a speech dictionary for speech synthesis, and the like in a memory or a hard disk. It is installed.

Reference numeral 807 denotes an IC card which stores, for example, an acoustic model for speech recognition translation from Chinese to Japanese, a word dictionary, a grammar dictionary, a translation example dictionary, a speech dictionary for speech synthesis, and the like in a memory or a hard disk. It is installed.

FIG. 9 is a block diagram showing the configuration of another embodiment of the voice / image recognition / translation apparatus according to the present invention. For example, the speech image recognition translator shown in FIG. 9 is a portable speech recognition translator, and even if it is a device composed of several LSIs such as a CPU, a memory, and a dedicated IC, it can be mounted on a semiconductor element. A configured chip may be used.

In FIG. 9, reference numeral 901 denotes a directional microphone for capturing voice, for example, voice for conversations exchanged at airports, train stations, airplanes, hotels, sightseeing spots, restaurants and shopping destinations of overseas travel destinations. .

Reference numeral 902 denotes a 16-bit analog / digital (A / D) conversion IC, in which a sound outside the audio band is removed by a filter or an amplifier in the microphone 901 and a continuous analog signal of audio data subjected to noise processing. Is sampled at a sampling frequency of audio, for example, 12 kHz, and is converted into a digital signal.

Reference numeral 903 denotes a voice capturing unit,
This is for performing serial / parallel conversion from serial data to parallel data on 16-bit audio data sampled by the D conversion IC 102 and temporarily storing the data in a register or the like.

Reference numeral 904 denotes a memory for storing voice data captured by the voice capturing unit 903, for example, continuous voice data for one phrase of conversation voice. It is a memory with a minimum capacity. Writing of the continuous audio data to the memory may be performed by software processing such as a CPU or by dedicated hardware.

A voice recognition processing unit 905 performs a series of voice recognition processes such as digital filtering, voice analysis, voice section detection, collation, and determination on continuous voice data written in the memory 904. Here, acoustic model data, dictionary data, and grammatical data necessary for speech recognition are registered and stored in a memory or the like in the speech recognition processing unit 905. The voice recognition processing may be performed by software processing such as a CPU or a DSP, or may be performed by dedicated hardware.

Reference numeral 906 denotes a high-resolution camera for taking in an image, for example, a CCD camera. This high-resolution camera 906 images the movement of the mouth of a person who generates this voice in accordance with the voice of a conversation exchanged at an airport, a station yard, an airplane, a hotel, a sightseeing spot, a restaurant or shopping, etc. at an overseas travel destination. Import as data.

Reference numeral 907 denotes a 16-bit analog / digital (A / D) conversion IC, which converts an analog signal from the CCD camera 906 into a digital signal by sampling at, for example, 12 kHz in synchronization with a sampling frequency of audio. I do.

Reference numeral 908 denotes an image reading unit.
This is for performing serial / parallel conversion from serial data to parallel data on 16-bit image data sampled by the D conversion IC 907 and temporarily storing the data in a register or the like.

Reference numeral 909 denotes a memory for storing image data fetched by the image fetching unit 908, for example, continuous image data for one phrase of conversation voice.
Further, the memory has a minimum necessary capacity for writing continuous image data. Writing of continuous image data to the memory can be performed by software processing such as CPU.
It may be performed by dedicated hardware.

Reference numeral 910 denotes an image recognition processing unit,
A series of image recognition processes such as digital filter, image conversion, binarization process, image analysis, feature extraction, collation, and determination are performed on the continuous image data written in step 09. Here, image model data, dictionary data,
The grammar data is registered and stored in a memory or the like in the image recognition processing unit 105. Image recognition processing is performed by CPU
Or by software processing such as a DSP, or by dedicated hardware. Here, the result of the image recognition processing is passed to the voice recognition processing unit.

Reference numeral 911 denotes a translation processing unit for performing a translation process on the speech speech recognition result output from the speech recognition processing unit 905 into a language to be translated. Voice recognition processing unit 9
The recognition result output from 05 is, for example, a text sentence of kana-kanji characters, such as nouns, particles, verbs, and adverbs in Japanese. In the translation process, the Kana-Kanji mixed sentence is subjected to syntax analysis and sentence generation from a dictionary, grammar rules, examples, and the like, and a translation result is output.

Reference numeral 912 denotes a speech synthesis processing unit that converts the translation result output from the translation processing unit 911 into a speech suitable for a conversation sentence and outputs the speech. In the speech synthesis processing unit 912, in order to make the conversation sentence speech more natural,
It also optimizes the pronunciation and accent of the words that make up the sentence, as well as the inflection of the entire sentence, synthesizes the speech of the conversation sentence, and performs processing for outputting natural speech that is easy to hear for the other party.

Reference numeral 913 denotes a 16-bit digital / analog (D / A) conversion IC.
The digital signal of the audio output from is transmitted through, for example, a low-pass filter (LPF) to an audio frequency band 12k.
Convert to analog signal at Hz.

Reference numeral 914 denotes a liquid crystal display (L) for displaying the progress of the speech recognition result and the translation result as text.
CD).

Reference numeral 915 denotes a speech recognition result, the progress thereof,
This is a speaker for synthesizing the translation result and outputting the speech.

A recognition result correction unit 916 corrects and corrects an erroneously recognized portion of the voice and image recognition results output from the voice recognition processing unit 905. If the translation result including the misrecognition part is immediately transferred and translated by the translation processing unit 911 as the recognition result of the conversation sentence to be translated, the translation result will be incorrect. Therefore, before translating the recognition result, the recognition result correcting unit 916 can improve the translation accuracy by correcting the voice conversation sentence to an appropriate sentence to be translated on its own.

[0102]

As described above, according to the present invention, it is possible to realize a speech recognition translator with high translation accuracy in the era of internationalization, and to provide a translation accuracy that assists mutual communication in conversation abroad at least while traveling overseas. A high portable speech recognition translator can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing one embodiment of a speech image recognition and translation apparatus according to the present invention.

FIG. 2 is an explanatory diagram showing image capturing and image processing in the voice image recognition device according to the present invention shown in FIG. 1;

FIG. 3 is a configuration diagram of an image capturing camera built-in microphone of the voice image recognition device according to the present invention shown in FIG. 1;

FIG. 4 is an explanatory diagram showing a voice and image capturing method and voice image recognition translation in the voice image recognition device according to the present invention shown in FIG. 1;

FIG. 5 is an explanatory diagram showing a voice and image capturing method of the voice and image recognition device according to the present invention shown in FIG. 1;

FIG. 6 is a flowchart showing a voice and image capturing method in the voice and image recognition device according to the present invention shown in FIG. 1;

FIG. 7 is an explanatory diagram showing a method of determining voice data in the voice image recognition device according to the present invention shown in FIG. 1;

FIG. 8 is an explanatory diagram showing an example in which the speech image recognition device according to the present invention shown in FIG. 1 is applied to a portable translator.

FIG. 9 is a block diagram showing another embodiment of the speech image recognition / translation apparatus according to the present invention.

FIG. 10 is a block diagram showing a configuration of a conventional portable speech translation device.

[Explanation of symbols]

101: microphone, 102: A / D conversion IC, 103: voice capturing unit, 104: memory, 105: voice recognition processing unit, 106: camera, 107: A / D conversion IC, 108
... Image capturing unit, 109, memory, 110, image recognition processing unit, 111, translation processing unit, 112, voice synthesis processing unit, 113, D / A conversion IC, 114, LCD, 115
... speakers.

Continued on the front page (51) Int.Cl. ⁶ Identification symbol FIG06F 15/38 A (72) Inventor Akio Amano 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo Inside the Central Research Laboratory, Hitachi, Ltd. (72) Inventor Isao Ito (Ii) Within the Semiconductor Division, Hitachi, Ltd. 5-2-1, Kamimizu Honcho, Kodaira City, Tokyo (72) Inventor Yuko Sato Within the Semiconductor Division, 5-2-1, Kamimizu Honmachi, Kodaira, Tokyo (72) Inventor Kazuka Ishiwata 5-2-1, Josuihoncho, Kodaira-shi, Tokyo Inside the Semiconductor Division, Hitachi, Ltd.

Claims (8)

[Claims] 1. A means for capturing voice and image, a memory for storing captured voice data, and a voice recognition processing unit for extracting a feature of the voice from a series of captured voice and performing voice recognition processing. A memory that stores the captured image data, an image recognition processing unit that extracts features of the captured image from a series of images, and performs an image recognition process;
A translation processing unit that translates the recognition result of the words and sentences recognized by the voice recognition and the image recognition into a word or a sentence to be translated, and the features of the speech that changes over time and the features of the image corresponding to the features of the speech A speech image recognition and translation device that recognizes a phrase of a sentence composed of a series of input voices and recognizes words constituting the sentence from the two correlations to recognize and translate the speech. When the feature of the mouth image indicates the end of a sentence phrase, it is determined that the sentence is composed of a series of input speeches, and the character or character string at the end of the phrase is recognized. An audio-visual recognition / translation device characterized by adding a mark to indicate. 2. A method according to claim 1, wherein the feature of a series of voices captured at the same sampling frequency and at the same time and voices extracted from the image changes from a consonant to a vowel, and further changes from a vowel to a silent state. In the process, if the features of the whole mouth corresponding to this voice feature and the image in the vicinity of the mouth indicate the end of the sentence phrase, it is determined that the sentence is composed of a series of input speech. Characteristic speech image recognition translator. 3. A feature of a series of Japanese voices and voices extracted from an image captured at the same time at the same sampling frequency, which changes from a consonant to a vowel, and further changes from a vowel to a silent state. In the process, if the features of the image of the whole mouth and its surroundings corresponding to the features of this voice indicate the end of the phrase of the sentence, it is determined that the phrase is a sentence composed of a series of input speech. And a speech image recognition translator for recognizing a particle at the end of the phrase. 4. A memory according to claim 1, wherein said memory for storing the fetched audio data is a memory for fetching sound data at all times, and a memory for fetching said audio data when a voice is inputted. And a memory (3) for constantly capturing image data and a memory (4) for capturing image data when audio is input, wherein the memory for storing the captured image data is configured; When it is determined that the voice has been input, the capture of the voice data and image data is started, and when it is determined that the voice input has been completed, the capture of the voice data and image data is stopped, and the stored voice data and image data are stored. An audio-visual recognition and translation apparatus characterized by performing speech recognition and translation for a speech. 5. A process according to claim 4, wherein the process of starting the capture of the voice data and the image data when the voice is determined to be input and the process of reading out the stored voice data and the image data are arbitrarily characterized in that the voice characteristics appear. At the set cycle (sampling frequency), the sound data stored in the memory (1) that always takes in the sound data is compared with the intensity Pi of the sound data taken at the time Ti and the time Ti− of the previous cycle. If the value of the difference ΔPi from the intensity Pi-1 of the sound data captured in 1 exceeds the arbitrarily set boundary value Pth of the intensity of the sound data, it is determined that the sound data is sound data. The next sound data is written into the memory (2) in order, and the image data is always taken in from the time when it is determined that the sound data is the sound data. When the writing of the image data to the memory (3) is completed, the next image data is sequentially written to the memory (4), and the audio data stored in the memory (1) and the audio data stored in the memory (2) are compared. Along with the image data stored in the memory (3) and the image data stored in the memory (4), the data is sequentially read in the direction of the time axis t to read out a series of audio waveform data. An audio-video recognition / translation apparatus characterized by forming image data indicating mouth movement. 6. A speech image according to claim 1, further comprising a recognition result correction unit for correcting or correcting the recognition result of the word or sentence recognized by the voice recognition and the image recognition. Recognition translation device. 7. A method according to claim 1, wherein the features of the image are extracted from a face image of a person, a whole mouth, and an image in the vicinity of the periphery thereof, which are normalized by performing resolution conversion and binarization processing of the image data. An audio-visual recognition / translation device characterized by the following. 8. A portable speech image recognition and translation apparatus according to claim 1, wherein said means for taking in the speech and the image is constituted by integrating a camera and a microphone.