JP2003058908A - Method and device for controlling face image, computer program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce quantity of data regarding the shape and to realistically realize lip sync animation by control of a face image in comparison with the conventional manner. SOLUTION: Pieces of first shape data regarding the shapes of a mouth when vowels are uttered are stored by every kind of vowels, kinds of consonants having common points in the shapes of the mouth when the consonants are uttered are sorted into the same groups, second shape data regarding the shapes of the mouth when the consonants sorted into the groups are uttered, are stored (#4) by every group, sounds of a language are sectioned (#5) by every vowel or consonant and motions of the face image are controlled (#8) on the basis of pieces of the first shape data corresponding to the vowels or pieces of the second shape data corresponding to the groups into which the consonants are sorted by every sectioned vowel and consonant.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a control method and control device for a so-called lip-sync animation of a face image whose mouth moves in accordance with a word.

[0002]

2. Description of the Related Art Conventionally, a technique has been proposed in which a mouth of a face image is moved according to a word so that the face image looks as if the word is spoken.

For example, there is a method of controlling a face image so that the mouth is simply opened and closed depending on the length of a word. With such a method, control of the face image is easy. However, the shape of the mouth is the same for all sounds, so it is not realistic.

Therefore, a lip synch animation (Lip Synchronizatio) that changes the shape of the mouth according to the words
n Animation) technology has been proposed. For example, according to the invention disclosed in Japanese Patent Publication No. 2000-507377, the shape of the mouth is determined according to the combination of two adjacent phonetic notations included in the word (word), in particular, the combination of a vowel and a consonant. To control.

Further, in order to realize more realistic animation, the Invers
A method of acquiring shape data by a method such as e Kinetics is used.

[0006]

[Problems to be Solved by the Invention] However, the special table 2000
According to the invention of No. 507377, the realism is improved as compared with the method of repeatedly opening and closing the mouth, but since the number of combinations of phonetic notations is large, the amount of shape-related data is large. Even when the Inverse Kinetics method is used, the amount of data regarding the shape is large because the shapes obtained from the video are very large.

When the amount of data increases, a higher speed network and a higher performance computer are required to realize the lip sync animation. Therefore, in order to expand the application range of lip sync animation, the amount of data required for lip sync animation must be reduced.

In view of such a problem, the present invention reduces the amount of data relating to the shape as compared with the conventional art, and can realize a lip sync animation by controlling a face image in a realistic manner. It is an object to provide a face image control device.

[0009]

A face image control method according to the present invention is a face image control method for controlling the movement of a face image so that the mouth moves in accordance with a word, and the vowel sounds are different for each kind of vowel. The first shape data relating to the shape of the mouth when uttering is stored, the types of consonants having common points in the shape of the mouth when pronounced are classified into the same group, and each group is classified into the group. The second shape data regarding the shape of the mouth when emitting a consonant is stored, the sound of the word is divided for each vowel or consonant, and for each vowel or consonant separated, the first corresponding to the vowel The operation of the face image is controlled based on the shape data or the second shape data corresponding to the group into which the consonants are classified.

Preferably, the method is a face image control method for controlling the operation of a face image so that the mouth moves in accordance with a word, and the first shape data regarding the shape of the mouth when the vowel is emitted for each kind of vowel. The first group having the characteristic that the mouth shape when memorizing and pronouncing the lips matches the second group having the characteristic that the mouth becomes slightly open, and the mouth of the phoneme immediately before that. The second shape data regarding the mouth shape of each characteristic is stored for each group of the third group having the feature that remains the shape, and the type of each consonant is set to the type of the mouth when the consonant is emitted. The shape is classified into the group having the closest characteristic among the respective characteristics, the sound of the word is divided for each vowel or consonant, and for each of the divided vowels or the consonant, the first corresponding to the vowel. Jo data or based on the second shape data corresponding to the group to which the consonant is classified for controlling the operation of the face image.

Alternatively, a predetermined number of consecutive 2 is added to the sound of the word.
Two consecutive vowels or consonants
One vowel or consonant is replaced with a subsequent vowel or consonant by a predetermined vowel or consonant, and the replaced sound is controlled.

Alternatively, the control may be performed so that the time for maintaining the shape of the mouth in the consonant is shorter than the time for maintaining the shape of the mouth in the vowel. Alternatively, when the sound of the word includes two predetermined consecutive vowels or consonants, the default time for keeping the mouth shape is changed for one or both of the two consecutive vowels or consonants. Then, the control may be performed.

The face image control apparatus according to the present invention stores the first shape data regarding the shape of the mouth when the vowel is emitted for each kind of the vowel, and the shape of the mouth when the vowel sounds is the same as that of the lips. For each group, a first group having a characteristic that becomes, a second group having a characteristic that becomes slightly open, and a third group having a characteristic that remains the mouth shape of the phoneme immediately before that. The second shape data regarding the mouth shape of each feature is stored, and the type of each consonant is classified into the group in which the mouth shape when the consonant is emitted has the closest feature among the features. Shape data storage means for storing as a vowel, consonant the sound of the word for each vowel or consonant, for each of the separated vowel and consonant, the first shape data or the consonant corresponding to the vowel Wherein corresponding to the class has been the group based on the second shape data having the operation control means for controlling operation of the face image.

A computer program according to the present invention is
The first shape data regarding the shape of the mouth when emitting the vowel for each type of vowel, and the consonant of the group for each type of consonant that has a common feature in the shape of the mouth when pronounced are emitted. A shape data storage means for storing second shape data relating to the shape of the mouth when,
A computer program used in a computer for controlling the movement of a facial image so that the mouth moves in accordance with words, the step of separating the sounds of the words into vowels and consonants, and the separated vowels and consonants A step of controlling the operation of the face image based on the first shape data corresponding to the vowel or the second shape data corresponding to the group into which the consonant is classified, Let The computer program is recorded on a recording medium according to the present invention.

[0015]

1 is a diagram for explaining the configuration of a face image control apparatus 1 according to the present invention, FIG. 2 is a diagram showing programs and data stored in a magnetic storage device 12, and FIG. 3 is a face image control. It is a figure explaining the functional composition of device 1.

As shown in FIG. 1, the face image control apparatus 1 is
Processing device 10, display device 11, magnetic storage device 1
2, a keyboard 13, a mouse 14, a microphone 15, a speaker 16 and the like.

The processing device 10 includes a CPU 10a and a RAM 1
0b, ROM 10c, various input / output ports 10d, various controllers 10e, and the like.
As shown in FIG. 2, the magnetic storage device 12 stores programs such as an operating system (OS) 12a, a face image control program 12b, and a modeling program 12c, and data used for various processes described later. ing.

The programs and data stored in the magnetic storage device 12 are loaded into the RAM 10b as needed. The loaded program is executed by the CPU 10a. The face image control apparatus 1 may be connected to another computer via the network 6N to download the program or data. Alternatively, the program or data may be loaded from various removable disks (recording media) such as the floppy disk 19a, the CD-ROM 19b, or the magneto-optical disk (MO) 19c.

The display device 11 includes a processing device 10
The processing result of is displayed. For example, the face image HF of the person is controlled so that the mouth moves in accordance with the input words,
The result of such control is displayed on the display screen HG of the display device 11 as a lip sync animation. The speaker 16 outputs the word as voice in accordance with the operation (lip sync animation) of the face image HF. This allows the user to recognize the face image HF as if he or she is speaking that word. The keyboard 13 or the microphone 15 is used for inputting words to be spoken in the face image HF.

As the face image control device 1, for example, a workstation or a personal computer is used. With such a configuration, the face image control device 1 includes, as shown in FIG.
Face model data storage unit 102, text data acquisition unit 1
03, voice data acquisition unit 104, voice text conversion unit 1
05, face image control unit 106, shape change data storage unit 10
7, shape group storage unit 108, and audio output unit 10
9 and the like are provided.

The face image HF of a person is obtained by projecting a three-dimensional shape model of the person in two dimensions from a predetermined direction. That is, the operation of the face image HF is controlled by deforming the three-dimensional shape model according to the words. [Preparation of 3D Geometric Model] FIG. 4 is a flowchart illustrating a flow of processing for generating a 3D geometric model, FIG. 5 is a diagram illustrating an example of a standard model DS, FIG. 6 is a flowchart illustrating a flow of deformation processing, FIG. 7 is a diagram schematically showing the surface S of the standard model DS and the points P of the three-dimensional measurement data, and FIG. 8 is a diagram for explaining a virtual spring for preventing abnormal deformation of the standard model DS.

The face model data generation unit 101 in FIG. 3 generates the three-dimensional shape model which is the basis of the face image HF described above. The generation of the three-dimensional shape model is performed by the procedure shown in the flowchart of FIG.

First, the standard model DS shown in FIG. 5 is roughly aligned with the three-dimensional measurement data of a person (for example, the user of the face image control apparatus 1) (# 101). The standard model DS is three-dimensional data having a standard face size and shape and structured around the entire circumference of the head. The three-dimensional measurement data is three-dimensional data of the user's face consisting of point clouds. That is, in step # 101, the standard model DS
The orientation, size, and position of the standard model DS are changed so that the distance between and the three-dimensional measurement data is minimized. Generally, as standard model DS and three-dimensional measurement data,
An expressionless state is used. The three-dimensional measurement data is prepared in advance by photographing the user with the three-dimensional measurement device.

Extract contours and characteristic points (# 10)
2). The contour and the feature points to be arranged at the same positions as the contour RK and the feature point TT for the standard model DS are
It is arranged on the three-dimensional measurement data or on the corresponding two-dimensional image.

As the characteristic points, for example, there are actually characteristic portions such as the ends of the eyes and mouth, the top of the nose, and the lower end of the chin, or there is no characteristic by itself such as the middle but the position. A part that is easy to identify is selected. As a contour,
The chin line, lip line, or eyelid line is selected.

In order to reduce the calculation amount and the error, data reduction is performed on the three-dimensional measurement data (# 10).
3). The standard model DS is modified (# 104). That is, an energy function defined in relation to the distance between each point of the three-dimensional measurement data and the surface of the standard model DS, or an energy function defined to avoid excessive deformation is used. Standard model D to minimize
Deform the surface of S.

Then, the target energy function and the control point are changed, and the processing for the change similar to step # 104 is repeated (# 105). Next, Step # 10
The transformation process of No. 4 will be described.

In FIG. 7, one of the point groups forming the three-dimensional measurement data is indicated by the point Pk. Standard model DS
On the surface S of, the point closest to the point Pk is indicated by Qk. The point Qk is an intersection when a perpendicular line is drawn from the point Pk to the surface S.

The method of fitting the surface S to the point cloud is as follows. Here, general fitting will be described. One point Pk in the point group, the corresponding point Qk, and the corresponding point group T = {(Pk, Qk), k =
1 ... n}, the fitting energy (Fittin
g Energy) Function Ff (U) is set as in the following equation (1).

[0030]

[Equation 1]

However, Qk (U) indicates that Qk is a function of U. In order to prevent the surface S from being excessively deformed, a virtual spring (elastic bar) KB shown in FIG. 8 is introduced. A stabilizing energy function for stabilizing the shape of the surface S is derived based on the constraint of the virtual spring KB.

That is, FIG. 8 shows a part of the surface (curved surface) S of the standard model DS to be fitted. The surface S has a control point group U = | ui, i = 1 ... n |
Is formed by. A virtual spring K is provided between the adjacent control points.
B is arranged. The virtual spring KB exerts a constraint by a pulling force between the control points and functions to prevent abnormal deformation of the surface S.

That is, when the distance between the adjacent control points u becomes large, the pulling force of the virtual spring KB becomes large accordingly. For example, when the point Qk approaches the point Pk and the distance between the control points u increases with the movement of the point Qk, the pulling force of the virtual spring KB increases. Point Q
Even if k is moved, if the distance between the control points u does not change, that is, if the relative positional relationship between the control points u does not change, the pulling force by the virtual spring KB does not change. Stabilization energy is defined as an average of the pulling force of the virtual spring KB over the entire surface S. Therefore, the stabilization energy increases when part of the surface S projects and deforms. If the entire surface S moves on average, the stabilization energy is zero.

The stabilized energy function Fs (U) is expressed by the following equation (2).

[0035]

[Equation 2]

Here,

[0037]

[Equation 3]

Are the initial end points of the virtual spring KB,
It is the end point of the virtual spring KB after deformation. c is a spring coefficient, and M is the number of virtual springs KB. Also, the following relationship holds.

[0039]

[Equation 4]

Therefore, when the spring coefficient c is increased,
The virtual spring KB becomes hard and difficult to deform. By introducing such a stabilizing energy function Fs (U), a constant constraint is provided for the shape change of the surface S,
Excessive deformation of the surface S can be prevented.

Using the fitting energy function Ff (U) and the stabilizing energy function Fs (U) described above, the fitting evaluation function F (U) is defined by the following equation (3).

F (U) = WfFf (U) + WsFs (U) (3) Here, Wf and Ws are weighting coefficients for normalization. The deformation of the surface S and the search for corresponding points are repeated so that the evaluation function F (U) of the equation (3) becomes sufficiently small,
Perform face fitting. For example, fitting is performed in the direction in which the derivative of F (U) with respect to U approaches 0.

In FIG. 6, in the transformation process, first, the point P
A point Qk corresponding to k is calculated and a pair of points Pk and Qk is created (# 111). Deform the surface S (# 11
2) Calculate the modified evaluation function F (U) (# 11
3). Until the evaluation function F (U) converges (Y in # 114
es), the process is repeated.

As a method of determining the convergence of the evaluation function F (U), the method of setting the convergence when the evaluation function F (U) becomes smaller than a predetermined value, and the change rate compared with the previous calculation are predetermined. It is possible to use a known method such as a method of converging when the value becomes less than or equal to the value.

By such processing, the standard model DS can be transformed to generate a three-dimensional shape model having the shape of the user's face. [Definition of Muscle] FIG. 9 is a diagram showing an example of the configuration of the face model 3M, FIG. 10 is a diagram showing an example of muscle placement data 71, and FIG.
Is a diagram showing an example of the node influence data 72, and FIG. 12 is a diagram explaining an example of a range affected by the movement of a certain node N.

The three-dimensional shape model of the user's face generated by the face model data generation unit 101 is stored in the face model data storage unit 102 shown in FIG. Hereinafter, the three-dimensional shape model will be referred to as “face model 3M”. The face model data storage unit 102 also stores the muscle placement data 71 shown in FIG. 10 and the node influence data 72 shown in FIG. 11.

In FIG. 9A, the intersections of a plurality of thin straight lines are the vertices (Model Vertex) V of the face model 3M.
Indicates. The position of the surface or skin of the face is determined by the constituent vertices V.

A thick straight line indicates an edge E which means a muscle of the face model 3M. A black circle indicates a node N (Node) which means an end point of a muscle. That is, the position of the muscle (edge E) is determined by two different nodes N.
The node N (N1, N2, ...) Corresponds to each control point u of the three-dimensional shape model, and is arranged at a position that is an end point of each muscle of the entire face. It should be noted that FIG.
9 to facilitate understanding of the relationship between the edge E and the edge E.
The apexes V are omitted from FIG. Figure 9
(A) and (b) are the node N and edge E of the right half of the face.
Although omitted, the node N and the edge E actually exist as in the left half.

The position of the node N is represented as the relative position of the constituent vertex V as in the following equation (4).

[0050]

[Equation 5]

As shown in FIG. 10, the muscle placement data 71 is data relating to the structure of each muscle (edges E1, E2, ...). The first parameter of the edge E indicates the two nodes N that are the end points of the edge E. The second parameter of the edge E indicates which end point (node N) is moved at what ratio (weight) when the edge E (muscle) is displaced. For example, edge E
The second parameter “0.7, 0.3” of 3 is the node N
It is indicated that 4 is moved at a ratio of 7 and node N3 is moved at a ratio of 3.

The position where the node N moves when the edge E is displaced is obtained by the following equation (5).

[0053]

[Equation 6]

However, since there are actually a plurality of nodes N related to the edges E, the position of the node N after movement can be obtained by a convergence calculation or a simultaneous calculation. As shown in FIG. 11, the node influence data 72 is data relating to the influence exerted on the constituent vertex V with the movement of the node N. The second parameter of the node influence data 72 indicates the constituent vertex V that is affected when each node N moves. That is, it shows the range of influence when the node N moves. The constituent vertices V affected by the movement of the node N are concentrated around the node N. For example, in FIG. 12, the constituent vertex V affected by the movement of the node N indicated by the large black circle is 9 indicated by the small black circle.
Two constituent vertices V.

The first parameter is the degree of influence (intensit) on the constituent vertex V when the node N moves.
y) is shown. If this value is large, the movement amount (displacement amount) of the constituent vertex V accompanying the movement of the node N becomes large.

As node N moves, constituent vertex V
The position to move is calculated by the following equation (6).

[0057]

[Equation 7]

[Data for controlling the shape of the face model] FIG. 13 shows a shape group to which vowels and consonants usually belong, FIG. 14 shows an example of the shape group data 75, and FIG. 15 shows shape change data 74. 16 shows an example of FIG.
FIG. 8 is a diagram showing an example of the shape of a face model when a sound belonging to each shape group is emitted.

In this embodiment, as shown in FIG. 13, five vowel groups (shape groups A, E, I, in order to reduce the amount of data relating to changes in the shape of the face model 3M).
O, U) and three groups (shape groups 1 to 3) for classifying each consonant type based on the characteristics of the mouth shape at the time of pronunciation.

To the shape groups A, E, I, O, and U, one type of vowel of "a", "e", "i", "o", and "u" belongs, respectively. Shape group 1 is a group of consonant sounds that are pronounced with lips, shape group 2 is a group of consonant sounds that are pronounced with the mouth in a predetermined shape without matching lips, and shape group 3 is a shape of the mouth of a sound that was emitted before. A group of consonants that are pronounced as they are. According to this classification, the shape group 1 usually includes five types of consonants “b, f, m, p, v”, and the shape group 2 includes “d, g, j, k, l,”.
11 types of consonants "n, r, s, t, w, z" belong to the shape group 3, and two types of consonants "h, y" belong to the shape group 3.

The shape change data storage unit 107 of FIG.
The shape change data 74 shown in FIG. The shape change data 74 includes the shape groups A, E, I, O, U, 1, 2
When the sound belonging to the face model 3M is emitted, the face model 3M has the shapes shown in FIGS. 16 (a) to 16 (g).
This is data for changing each muscle of M (edges E1, E2, ...). Each value of the shape change data 74 indicates the degree of muscle contraction. The state of non-contraction is "0", and the most contracted state is "20". For example,
If the amount of deformation (degree of contraction) is “15.0”, it means that the muscle (edge E) contracts by 75%. In the case of the shape group 3, since the face model 3M is kept in the shape of the sound (phoneme) emitted previously, it does not have the value of the deformation amount.

To control the shape of the face model 3M, which will be described later, the words to be spoken by the face model 3M are divided into phonemes and decomposed, and the edge E is generated based on the shape change data 74 corresponding to the shape group to which each phoneme belongs. Is done by changing. For example, in the case of the word "kan", the face model 3M is divided into "k, a, n" and the shape groups 2, A, 2 are formed in order.
To control.

However, in general, words may not be pronounced as described (spelled) due to the influence of the immediately preceding sound (phoneme). For example, in the case of the word "kou", "u" is affected by "o" immediately before it, and "ko" is affected.
pronounced "o". At this time, the shape of the mouth is also shown in FIG.
The shape is not the “u” shape shown in (e) but the “o” shape shown in FIG. 16 (d).

As shown in FIG. 14, the shape group data 75 stored in the shape group storage unit 108 is pronounced with an unusual mouth shape under the influence of the immediately preceding phoneme as in the above example. This is data about phonemes. Specifically, the shape group data 75 indicates to which shape group each phoneme is changed when influenced by the phoneme immediately before it. For example, "g" is usually shape group 2
However, when there is "n" immediately before, it is changed to shape group 3 as shown in bold. [Control of Face Image (Execution of Animation)] Next, control of operation of the face image HF of the user will be described.

FIG. 17 is a diagram showing an example of a shape group to which each phoneme of "kouminkan" belongs, FIG. 18 is a diagram explaining the duration of vowels, consonants, and continuous vowels, and FIG. 19 is a diagram showing an example of a time table. Is.

The text data acquisition unit 103 in FIG. 3 acquires, as the text data 73, the words that are the basis for deforming the shape of the face model 3M (that is, the words that cause the face model 3M to speak). For example, the character string input by the user operating the keyboard 13 is acquired as the text data 73. Alternatively, a message written in an electronic mail received from another user may be used as the text data 73.

Words may be input by voice instead of text data. The voice data acquisition unit 104 converts the voice of the user who speaks into the microphone 15 into the voice data 74.
To get as. The voice data 74 is converted into text data 73 by the voice / text converter 105.

The face image control unit 106 includes the shape acquisition unit 16
1. The time distribution unit 162, the shape interpolation unit 163, and the moving image generation unit 164 control the face image HF to operate in accordance with the input words.

The shape acquisition unit 161 uses the text data 73.
Is divided into phonemes, that is, vowels or consonants, and decomposed. For example, when the text data 73 is “public hall”, “k, o, u, m, i, n,
It is divided into nine phonemes “k, a, n”. Then, the shape group data 7 shown in FIG. 14 and the phoneme immediately before it is determined to which shape group each segmented phoneme belongs.
Calculate based on 5. That is, for each phoneme, it is determined based on the shape group data 75 whether or not the shape group is changed under the influence of the immediately preceding phoneme. For example,
As shown in FIG. 17, the third “u” is immediately preceded by an “o”.
Therefore, the shape group is changed to O. The other phonemes are not affected by the immediately preceding phoneme, and thus remain in the normal shape group. In this way, as shown in FIG. 17, the face model 3 is matched to each phoneme “k, o, ..., N”.
A shape group for controlling the shape of M is determined.
However, in FIG. 17, “G1” and “G2” represent shape groups 1 and 2, respectively, and “A”,
“I” and “O” are shape groups A, I, and
And O are shown.

The time allocation section 162 sets the timing for deforming the shape of the face model 3M in accordance with each phoneme. The setting is performed based on the following rules.

As shown in FIG. 18, the time from a certain shape (for example, an expressionless shape) to the change of the next phoneme (rise time T1) and the time until the shape returns to the expressionless shape. The time (end time T3) is set to an extremely short time, for example, 0.1 second or less. At this time, T
It may be 1 = T3.

Time to maintain the shape of vowel (duration T
b2), the time for maintaining the shape of the consonant (duration Ts
The relationship between 2) and the length of the time (duration Tr2) for maintaining the shape of a vowel (continuous vowel) when the immediately preceding phoneme is a vowel is Ts2 <Tr2 <Tb2. At standard conversation speed, Ts2 is about 0.1 seconds,
Tb2 is about 0.4 seconds. The length of these times can vary depending on the speaking speed.

The timing of two adjacent phonemes is shown in FIG.
As shown in 9, when the previous phoneme ends (t = tb)
Set so that the subsequent phoneme rise is completed. That is, the subsequent phonemes are arranged so as to start rising before the end time T3 before tb (t = ta).

According to the above rule, for example, "koom
The shape group corresponding to each phoneme of "inkan" is shown in FIG.
It is arranged like a timetable shown in FIG. Returning to FIG. 3, the shape interpolating unit 163 interpolates the data of the shape of the face model 3M when moving from one phoneme to the next phoneme. That is, the shape of the face model 3M at the portion where the rise time of one phoneme and the end time of the other overlap is linearly approximated for simplicity, and the position of each constituent vertex V of the face model 3M is calculated based on the following equation (7). Ask for.

[0075]

[Equation 8]

The moving image generation section 164 has a shape acquisition section 161.
And the face model 3M obtained by the shape interpolation unit 163
The face image HF is operated by projecting the shape, that is, the position of the constituent vertex V according to the time table shown in FIG. 19 in a two-dimensional manner from a predetermined direction, and a lip sync animation is generated.

The voice output unit 109 converts the words into voices and outputs them in synchronization with the lip sync animation. For example, when a predetermined phoneme rises, the face image control unit 106
The sound is sequentially output in accordance with the signal (trigger) emitted from. A known speech synthesis technique is used as a method for converting text data into speech.

Next, the flow of processing for moving the user's face image HF in accordance with the input words to generate a lip sync animation will be described with reference to the flowchart. 20 is a flowchart for explaining the overall processing flow of the face image control apparatus 1, FIG. 21 is a flowchart for explaining the shape acquisition processing flow, and FIG. 22 is a flowchart for explaining the time allocation processing flow.

As shown in FIG. 20, first, the face model 3M which is the basis of the face image of the user is generated (# 1), and the edge E is generated.
Set the (muscle) position (# 2). The range of the constituent vertex V affected by the change in the position of the node N is set, and the effect on the skin when the muscle of the face model 3M moves is set (# 3). The position of the edge E is set for each shape group (# 4). Steps # 1 to # 4 are preparations for generating a face image, and may be omitted if the face model 3M has already been generated and each setting is performed.

The words to be spoken on the face image of the user are input as text data (# 5). If input by voice, convert it to text. The shape of the face model 3M is acquired for each phoneme included in the input text data (#
6). That is, as shown in FIG. 21, the text data is divided into phonemes and decomposed (# 121), and the shape group to which each phoneme belongs is obtained based on the shape group data 75 shown in FIG. 14 (# 122). The shape of the face model 3M corresponding to each shape group is acquired based on the shape change data 74 shown in FIG. 15 (# 123).

Each decomposed phoneme is placed in the timetable (# 7). That is, as shown in FIG. 22, when the phoneme is a consonant (No in # 131), the duration of the phoneme is set to Ts2 (# 132), and when it is a continuous vowel (Yes in # 133), the duration is Let Tr2 (# 13
4) If it is a normal vowel (No in # 133), the duration is set to Tb2 (# 135).

The shape is interpolated with respect to the overlapping portion of the adjacent phonemes arranged in the timetable (#
8). Then, the shape of the face model 3M is deformed based on the data obtained by the above processing, and a lip sync animation is generated (# 9).

According to the present embodiment, the sound of a word is divided for each phoneme, that is, for each vowel or consonant, and the operation of the face image is controlled for each divided phoneme. It is possible to realize various lip sync animations. Furthermore, in this embodiment,
Five vowel shape groups and three consonant shape groups are provided, and the consonant type is classified into any of the three consonant shape groups. Therefore, since it is only necessary to prepare eight types of data regarding the shape, it is possible to further reduce the amount of data.

In this embodiment, the consonant duration T
s2 is set to be shorter than the duration Tb2 of the vowel, and the change in the shape of the face image is controlled. As a result, a more realistic lip sync animation can be realized. Further, since the shape data is interpolated between two adjacent phonemes, the change in shape between phonemes is integrated, and a lip-sync animation that smoothly changes as a whole can be realized.

In the present embodiment, the face model is obtained by fitting the standard model to the user's three-dimensional measurement data, but the face model may be obtained by fitting the standard model to the user's two-dimensional image. Or
The face model may be created using various computer graphics (CG) programs.

Not only may the mouth of the face image be moved in accordance with the words, but the eyes, eyebrows, or facial expression may be changed. For example, data representing changes in eyes, eyebrows, or facial expressions with time is prepared, and the muscle (edge E) of the face model is controlled based on the data.

The face image control apparatus 1 of this embodiment can be used as a communication means between a plurality of users. For example, when the user α and the user β have a conversation on a mobile phone having an image processing function, the users α and β
Sends his face model to the other party's mobile phone device in advance. The mobile phone terminal of the user β divides the received voice of the user α into phonemes, and performs steps # 6 to # shown in FIG.
The process of 9 is performed, and the lip sync animation of the user α is displayed. Similarly, the mobile phone terminal of the user α displays the lip sync animation of the user β.

In this embodiment, the face image is controlled using the three-dimensional face model, but the two-dimensional face model may be used. In the present embodiment, the consonant types are classified into three shape groups, but may be classified into four or more shape groups. Further, in the case of English or the like, text data may be converted into phonetic symbols and the face image may be controlled based on the phonetic symbols.

In addition, the configuration, processing content, processing order, etc. of the whole face image control apparatus 1 or each part can be appropriately changed in accordance with the spirit of the present invention.

[0090]

According to the present invention, it is possible to reduce the amount of data relating to the shape as compared with the related art and to realize a lip sync animation realistically by controlling a face image.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a configuration of a face image control apparatus according to the present invention.

FIG. 2 is a diagram showing programs and data stored in a magnetic storage device.

FIG. 3 is a diagram illustrating a functional configuration of a face image control device.

FIG. 4 is a flowchart illustrating a flow of processing for generating a three-dimensional shape model.

FIG. 5 is a diagram showing an example of a standard model.

FIG. 6 is a flowchart illustrating the flow of a transformation process.

FIG. 7 is a diagram schematically showing a surface S of a standard model and a point P of three-dimensional measurement data.

FIG. 8 is a diagram for explaining a virtual spring for preventing abnormal deformation of the standard model.

FIG. 9 is a diagram showing an example of the configuration of a face model.

FIG. 10 is a diagram showing an example of muscle placement data.

FIG. 11 is a diagram showing an example of node influence data.

FIG. 12 is a diagram illustrating an example of a range affected by the movement of a certain node.

FIG. 13 is a diagram showing shape groups to which vowels and consonants usually belong.

FIG. 14 is a diagram showing an example of shape group data.

FIG. 15 is a diagram showing an example of shape change data.

FIG. 16 is a diagram showing an example of the shape of a face model when a sound belonging to each shape group is emitted.

FIG. 17 is a diagram showing an example of a shape group to which each phoneme of “kouminkan” belongs.

FIG. 18 is a diagram illustrating durations of vowels, consonants, and continuous vowels.

FIG. 19 is a diagram showing an example of a time table.

FIG. 20 is a flowchart illustrating the overall processing flow of the face image control apparatus.

FIG. 21 is a flowchart illustrating a flow of a shape acquisition process.

FIG. 22 is a flowchart illustrating a processing flow of time allocation.

[Explanation of symbols]

1 Face image control device 106 face image control unit (motion control means) 107 Shape Change Data Storage Unit (Shape Data Storage Means) 19a to 19c Removable disk (recording medium) 74 Shape change data (shape data) HF face image

Claims (9)

[Claims] 1. A face image control method for controlling the motion of a face image so that the mouth moves in accordance with a word, wherein first shape data relating to the shape of the mouth when the vowel is emitted for each type of vowel. The second shape data on the shape of the mouth when memorizing and classifying consonant types that have common points in the shape of the mouth when pronounced, into the same group, and emitting consonants classified into the group for each group Storing the sound of the word for each vowel or consonant, and for each of the separated vowels or consonants, the first shape data corresponding to the vowel or a group corresponding to the group in which the consonant is classified. A face image control method comprising: controlling the operation of the face image based on the second shape data. 2. A face image control method for controlling the movement of a face image so that the mouth moves in accordance with a word, wherein first shape data relating to the shape of the mouth when the vowel is emitted for each type of vowel. The mouth shape when memorized and pronounced is such that the first group has the characteristics that make the lips fit together, the second group that has the characteristics that makes the mouth slightly open, and the mouth shape of the phoneme immediately before that. The second shape data regarding the mouth shape of each characteristic is stored for each group of the third group having the remaining characteristics, and the type of each consonant is defined as the mouth shape when the consonant is emitted. Is classified into the group having the closest feature among the features, the sound of the word is divided into vowels or consonants, and each of the separated vowels or consonants has the first shape corresponding to the vowel. Day Or based on the second shape data corresponding to the group to which the consonant is classified for controlling the operation of the face image, a face image control method, characterized in that. 3. When the sound of the word includes two predetermined consecutive vowels or consonants, the vowel or consonant following the two consecutive vowels or consonants is replaced with a predetermined vowel or consonant, and replaced. The face image control method according to claim 1, wherein the control is performed on the generated sound. 4. The face image control method according to claim 1 or 3, wherein the control is performed such that the time for maintaining the mouth shape during consonants is shorter than the time for maintaining the mouth shape during vowels. . 5. When the sound of the word includes two predetermined continuous vowels or consonants, one of the two continuous vowels or consonants or both of them has a predetermined mouth shape. The face image control method according to claim 1, wherein the control is performed by changing the time. 6. A face image control device for controlling the movement of a face image so that the mouth moves in accordance with a word, the first shape data relating to the shape of the mouth when the vowel is emitted for each type of vowel, And shape data storage means for storing second shape data relating to the shape of the mouth when emitting the consonant of the group for each group in which the types of consonants that have a common point in the shape of the mouth when sounding, The sound of a word is divided for each vowel and each consonant, and for each of the divided vowels and the consonants, the first shape data corresponding to the vowel or the second corresponding to the group in which the consonant is classified A face image control device comprising: an operation control unit that controls the operation of the face image based on shape data. 7. A face image control device for controlling the movement of a face image so that the mouth moves in accordance with a word, wherein first shape data regarding the shape of the mouth when the vowel is emitted for each type of vowel. The mouth shape at the time of memorizing and pronouncing is such that the first group has a feature that makes the lips fit together, the second group has a feature that makes the mouth slightly open, and the mouth shape of the phoneme immediately before that. The second shape data regarding the mouth shape of each characteristic is stored for each group of the third group having the remaining characteristics, and the type of each consonant is the shape of the mouth when the consonant is emitted. Shape data storage means for classifying and storing into the group having the closest feature among the features, separating the sound of the word for each vowel or consonant, and for each of the separated vowels and consonants, the vowel Motion control means for controlling the motion of the face image based on the corresponding first shape data or the second shape data corresponding to the group into which the consonants are classified. Face image control device. 8. The first shape data relating to the shape of the mouth when the vowel is emitted for each kind of the vowel, and the group of the consonant having the common point in the shape of the mouth when the vowel is pronounced. A computer program used for a computer that has shape data storage means for storing second shape data relating to the shape of the mouth when a consonant of a group is emitted, and controls the operation of a face image so that the mouth moves in accordance with a word. There is a step of dividing the sound of the word into vowels and consonants, and for each of the separated vowels and the consonants, the first shape data corresponding to the vowel or the group into which the consonant is classified A step of controlling the operation of the face image based on the corresponding second shape data; Program. 9. A computer-readable recording medium in which the computer program according to claim 8 is recorded.