CN113344770B - Virtual model and construction method thereof, interaction method and electronic device - Google Patents

Abstract

The present application discloses a virtual model and a construction method thereof, an interaction method, and an electronic device, the method comprising obtaining a three-dimensional model of a target person; performing parameterization processing on the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model; performing transformation processing on each point in the standard model based on the target transformation matrix to obtain a transformed standard model; and editing the transformed standard model to obtain a virtual model, wherein the virtual model contains a plurality of expression target bodies. The method provided by the present application uses a standard model to approach the three-dimensional model of the target person, and obtains a three-dimensional virtual model by editing the standard model (the number of point clouds, the number of facets, etc. are relatively fixed and evenly distributed), with low cost.

Description

Virtual model and construction method thereof, interaction method and electronic device

Technical Field

The present application relates to the field of human-computer interaction, and further to a virtual model and a construction method thereof, an interaction method and an electronic device.

Background Art

With the development of artificial intelligence, human-computer interaction has gradually been more and more widely used. For example, in some service industries, voice robots are used to interact with users to save labor costs and improve service efficiency.

However, the human-machine interaction between voice robots and users is generally voice interaction, which lacks visual intuitiveness and poor user experience. Some current visual interaction methods generally use 2D information in video streams (such as two-dimensional images, etc.) to interact with users, which lacks realism compared to 3D information (such as three-dimensional digital human models, etc.).

The three-dimensional digital human models in the prior art are generally obtained by modelers using modeling software. Not only is the modeling cost high, but the quality of the model is also related to the experience of the modeler. The three-dimensional digital human models obtained by different modelers vary greatly, which is not conducive to practical applications.

Summary of the invention

One advantage of the present invention is that it provides a virtual model and a construction method thereof, an interaction method and an electronic device, wherein the method can construct a three-dimensional virtual model to achieve human-computer interaction without manual modeling and at a low cost.

In a first aspect, an advantage of the present invention is to provide a method for constructing a virtual model, comprising:

Obtain a three-dimensional model of the target person;

Parameterizing the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model;

Transforming each point in the standard model based on the target transformation matrix to obtain a transformed standard model;

The transformed standard model is edited to obtain a virtual model, wherein the virtual model includes a plurality of expression target bodies.

In one possible implementation, the parameterization of the three-dimensional model using the standard model to obtain a target transformation matrix for each point in the standard model includes:

Obtaining a first error term according to the constraint of each adjacent point in the three-dimensional model;

Obtaining a second error term according to the constraints of the transformation matrix of each point in the standard model;

Obtaining a third error term according to constraints on corresponding points in the standard model and the three-dimensional model;

A target transformation matrix is determined according to a weighted minimum value of the first error term, the second error term, and the third error term.

In one possible implementation, the first error term is given by the formula

Calculated, where E _S is the first error term, _Ti and T _j are transformation matrices,

The second error term is given by the formula

Calculated, where E _I is the second error term,

The third error term is given by the formula

Calculated, where _EC is the third error term, _Vi is the coordinate of the i-th point in the standard model after transformation by the transformation matrix, Ci is the coordinate of the i-th point in the three-dimensional model corresponding to the point Vi in the standard model,

The weight and minimum value are given by the formula

Calculated, where W _S , W _I , and W _C are weight values.

In one possible implementation, editing the transformed standard model to obtain a virtual model includes:

In response to the user's operation of editing the expression target body, a plurality of the expression target bodies are constructed in the transformed standard model to obtain a virtual model.

In a second aspect, the present application provides a virtual model interaction method, comprising:

Get interactive information input by the user;

Based on the interaction information, a video stream is acquired, wherein the video stream includes a plurality of pictures;

Extracting facial features of the person from each of the pictures;

Based on the facial features of the character and the virtual model, acquiring expression coefficients of a plurality of expression targets in the virtual model, wherein the virtual model is obtained by the method according to the first aspect;

Based on the expression coefficient of each of the expression targets, the virtual model is driven, wherein the driven virtual model is dynamically displayed.

In one possible implementation, extracting facial features of a person from each of the pictures includes:

Acquire the collected facial image of the person from each of the pictures;

Feature extraction is performed on the character's facial image to obtain the character's facial features.

In one possible implementation, extracting facial features of a person from each of the pictures includes:

Acquire the collected voice signal from each of the pictures;

The mouth shape of each word in the speech signal is recognized to obtain the facial features of the person.

In one possible implementation, there is a correspondence between feature points in the facial features of the person and key points in the virtual model, and acquiring expression coefficients of multiple expression targets in the virtual model based on the facial features of the person and the virtual model includes:

The expression coefficient of each expression target body is determined according to the positional relationship between the feature points and the key points.

In one possible implementation, the virtual model after driving is represented by the formula

It is calculated, where Base is the virtual model, _Bi is the i-th expression target body, _Wi is the expression coefficient of the i-th expression target body, and B is the virtual model after driving.

In a third aspect, the present application provides a virtual model construction device, comprising:

A model acquisition module is used to acquire a three-dimensional model of a target person;

A processing module, used for performing parameterization processing on the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model;

A transformation module, used for transforming each point in the standard model based on the target transformation matrix to obtain a transformed standard model;

The editing module is used to edit the transformed standard model to obtain a virtual model, wherein the virtual model includes a plurality of expression target bodies.

In a fourth aspect, the present application provides an interactive device for a virtual model, comprising:

An interactive information acquisition module, used to acquire interactive information input by a user;

A video stream acquisition module, used to acquire a video stream based on the interaction information, wherein the video stream includes multiple pictures;

An extraction module, used for extracting facial features of a person from each of the pictures;

An expression coefficient acquisition module, used to acquire expression coefficients of multiple expression targets in the virtual model based on the facial features of the character and the virtual model, wherein the virtual model is obtained by the method described in the first aspect;

The driving module is used to drive the virtual model based on the expression coefficient of each expression target body, wherein the driven virtual model is dynamically displayed.

In a fifth aspect, the present application provides a virtual model, which is constructed by the method described in the first aspect, and the virtual model is used for display in an electronic device.

In a sixth aspect, another advantage of the present invention is to provide an electronic device, comprising:

One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, cause the device to perform the method as described in the first aspect or the second aspect.

In the seventh aspect, another advantage of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect or the second aspect.

In an eighth aspect, the present application provides a computer program, which, when executed by a computer, is used to execute the method described in the first aspect or the second aspect.

In one possible design, the program in the eighth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing a method for constructing a virtual model according to an embodiment of the present invention.

FIG. 2 is a schematic diagram showing an embodiment of a method for interacting with a virtual model of the present invention.

FIG3 shows a schematic structural diagram of an embodiment of a device for constructing a virtual model of the present invention.

FIG. 4 shows a schematic structural diagram of an embodiment of an interactive device for a virtual model of the present invention.

FIG. 5 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

Application Overview

One of the existing human-computer interaction methods is to use a voice robot to interact with the user. This method is a voice interaction method, lacks visual intuitiveness, and has a poor user experience. Some current visual interaction methods generally use 2D information (such as two-dimensional images, etc.) in the video stream to interact with the user, which lacks realism compared to 3D information (such as three-dimensional digital human models, etc.).

However, the three-dimensional digital human models in the prior art are generally obtained by modelers using modeling software. Not only is the modeling cost high, but the quality of the model is also related to the experience of the modeler. The three-dimensional digital human models obtained by different modelers vary greatly, which is not conducive to practical applications.

The applicant has found that since the three-dimensional model obtained by 3D scanning has a huge number of point clouds, grids, and maps, the number is not fixed and the distribution is uneven, the three-dimensional model obtained by 3D scanning is difficult to edit, and thus it is difficult to change the shape of the three-dimensional model, which is not conducive to constructing different expression targets in the three-dimensional model, such as smiling, opening the mouth, blinking, etc., making it difficult to drive the three-dimensional model and not conducive to achieving human-computer interaction.

Therefore, the present application provides a method for constructing a virtual model, which may include: obtaining a three-dimensional model of a target person; parameterizing the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model; transforming each point in the standard model based on the target transformation matrix to obtain a transformed standard model; editing the transformed standard model to obtain a virtual model, wherein the virtual model contains multiple expression target bodies. It can be seen that the method provided by the present application uses a standard model to approach the three-dimensional model of the target person, and obtains a three-dimensional virtual model by editing the standard model (the number of point clouds, the number of facets, etc. are relatively fixed and evenly distributed). Therefore, the editing difficulty is relatively low (there is no need to edit the three-dimensional model of the target person), and it is conducive to constructing different expression target bodies in the model, and it is conducive to driving the virtual model to achieve human-computer interaction, and there is no need for manual modeling, and the cost is relatively low.

Exemplary virtual model construction method

Referring to FIG. 1 , a method for constructing a virtual model according to an embodiment of the present invention is shown. The three-dimensional virtual model constructed by the method can be displayed on electronic devices such as computers, mobile phones, smart watches, smart robots, smart homes, cars, etc., to achieve human-computer interaction.

As shown in FIG. 1 , the method may include:

S101: Obtain a three-dimensional model of a target person.

In this embodiment, a 3D scanner is used to perform a 360° 3D scan on a target person (such as a natural person) to obtain a three-dimensional model of the target person. Preferably, the three-dimensional model may include a point cloud, a number of triangular meshes, and a number of textures. It is understandable that the method may also include: performing point cloud denoising, multi-frame alignment, or triangulation processing on the three-dimensional model of the target person obtained by scanning to obtain a processed three-dimensional model.

S102: Parameterize the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model.

In step S102, the parameterization process refers to transforming each point in the standard model using a transformation matrix so that the standard model approaches the three-dimensional model, and determining a target transformation matrix for each point in the standard model according to the degree of proximity between the standard model and the three-dimensional model. Preferably, when the standard model is closest to the three-dimensional model, the corresponding transformation matrix is the target transformation matrix.

That is to say, after each point in the standard model is transformed according to the target transformation matrix, the transformed standard model is closest to the three-dimensional model.

It is understandable that the number of point clouds and facets in the standard model is relatively fixed and evenly distributed. The standard model may be a three-dimensional character standard model, and the standard model is not unique according to different needs of users.

In one possible implementation, step S102 may include:

S201, obtaining a first error term according to the constraint of each adjacent point in the three-dimensional model;

S202, obtaining a second error term according to the constraints of the transformation matrix of each point in the standard model;

S203, obtaining a third error term according to constraints of corresponding points in the standard model and the three-dimensional model;

S204: Determine a target transformation matrix according to a minimum value of a weighted sum of the first error term, the second error term, and the third error term.

In one possible implementation, for example, it is assumed that the transformation matrix of each point in the standard model is Ti (such as a 4*4 square matrix).

Then, the first error term is given by the formula (or the first optimization function)

Calculated, where E _S is the first error term, _Ti and T _j are transformation matrices,

The second error term is given by the formula (or second optimization function)

Calculated, where E _I is the second error term,

The third error term is given by the formula (or third optimization function)

Calculated, where _EC is the third error term, _Vi is the coordinate of the i-th point in the standard model after transformation by the transformation matrix, Ci is the coordinate of the i-th point in the three-dimensional model corresponding to the point Vi in the standard model,

The weight and minimum value are given by the formula

Calculated, where W _S , W _I , and W _C are weight values.

It can be understood that by substituting the first error term, the second error term and the third error term corresponding to the minimum value of the weighted sum into the above formula, the target transformation matrix can be obtained.

Those skilled in the art will appreciate that, when implementing the same principle or function, the calculation formula in the above parameterization process may not be unique and is not limited here.

S103. Transform each point in the standard model based on the target transformation matrix to obtain a transformed standard model.

That is to say, each point in the standard model is transformed using the target transformation matrix to change the coordinate value of each point in the standard model, thereby obtaining a transformed standard model.

It is understandable that, since the number of point clouds, the number of facets, etc. in the standard model is relatively fixed and evenly distributed, the standard model is easier to edit, which is conducive to constructing different expression targets in the model.

S104, editing the transformed standard model to obtain a virtual model, wherein the virtual model includes a plurality of expression targets.

In this embodiment, the expression target can be used to drive the virtual model to change the shape of the virtual model so that the virtual model can show corresponding expressions, such as blinking, smiling or opening the mouth.

For example, the formula for driving the virtual model using the expression target is as follows:

Among them, Base is the virtual model, _Bi is the i-th expression target body, _Wi is the expression coefficient of the i-th expression target body, and B is the virtual model after driving.

That is to say, as the expression coefficient of each expression target changes, the virtual model is driven to make corresponding changes, such as showing corresponding expressions, etc., which is conducive to achieving human-computer interaction.

In one possible implementation, step S104 may include:

In response to the user's operation of editing the expression target body, a plurality of the expression target bodies are constructed in the transformed standard model to obtain a virtual model.

That is to say, the user can edit the target objects such as smiling expressions, open mouth expressions or blinking expressions made in the editing software into the standard model to obtain the edited standard model, that is, the virtual model.

In summary, the method for constructing the virtual model provided in the present application uses a standard model to approach the three-dimensional model of the target person, and obtains a three-dimensional virtual model by editing the standard model (the number of point clouds, the number of facets, etc. are relatively fixed and evenly distributed). Therefore, the editing difficulty is relatively low (there is no need to edit the three-dimensional model of the target person), and it is conducive to constructing different expression target bodies in the model, which is conducive to driving the virtual model to achieve human-computer interaction, and there is no need for modeling, so the cost is low.

It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

Exemplary virtual model interaction method

As shown in Figure 2, the present application implements a virtual model interaction method, which is applied to electronic devices (such as computers, mobile phones, smart watches, smart robots, etc.). The virtual model can imitate the expressions of people in video streams, thereby realizing human-computer interaction.

Specifically, the interaction method may include:

S301: Acquire interaction information input by a user.

That is to say, the interaction information may include text information input by the user through a keyboard or a mouse, or a user voice collected in real time, or a user image collected in real time, etc.

S302, acquiring a video stream based on the interaction information, wherein the video stream includes a picture;

In this embodiment, the video stream is pre-stored in the electronic device, and the corresponding video stream is retrieved from the electronic device by identifying the interaction information, and the video stream contains information matching the user's interaction information for human-computer interaction. The video stream may contain multiple pictures, and the pictures may contain facial images of people or voice signals, etc. For example, in step S302, by performing voice recognition on the interaction information (such as the voice signal sent by the user), the target video stream can be extracted from the multiple video streams stored in the electronic device according to the recognition result, as the target video stream for human-computer interaction, such as the target video can be used to respond to the voice signal sent by the user, etc.

S303: Acquire facial features of the person from each of the pictures.

In this embodiment, the facial features of the person may include facial features of the person in the picture, so as to indicate the current facial expression state of the person in the picture, such as happy, confused, angry, etc. facial states.

In one possible implementation, step S303 may include:

S401, acquiring a collected facial image of a person from each of the pictures;

S402: extracting features from the facial image of the person to obtain facial features of the person.

For example, the facial image of the person may be an image captured by a camera, or a facial image of the person extracted from a video stream, etc., so as to extract facial features of the person from the facial image of the person. The facial features of the person may include feature points extracted from the facial image of the person, such as the coordinates or depth information of the feature points.

In one possible implementation, step S303 may include:

S501, acquiring a collected voice signal from each of the pictures;

S502: Recognize the mouth shape of each word in the voice signal to obtain facial features of the person.

For example, the voice signal may be a voice signal extracted from a picture in a video stream, or a voice signal collected by a voice sensor, etc., and a voice recognition model is used to recognize the mouth shape of each word in the collected voice signal to obtain facial features of the character.

S304: Based on the facial features of the character and the virtual model, acquiring expression coefficients of a plurality of expression targets in the virtual model.

In this embodiment, the virtual model is obtained by the construction method provided by the method embodiment shown in FIG1 . The specific steps or principles thereof can refer to the method embodiment shown in FIG1 above, and will not be described in detail here.

Preferably, there is a corresponding relationship between the feature points in the facial features of the person and the key points in the virtual model, for example, the corners of the eyes (feature points) in the facial features of the person correspond to the corners of the eyes (key points) in the virtual model.

In one possible implementation, step S304 may include:

The expression coefficient of each expression target body is determined according to the positional relationship between the feature points and the key points.

That is to say, the expression coefficient of each expression target body can be calculated by using the acquired positional relationship (such as coordinate difference, etc.) between the feature points of the character's facial features and the key points in the virtual model.

In this embodiment, the expression coefficient of each expression target body is obtained by optimizing the optimization function using the constraints of the coordinates or positions of the key points projected from the virtual model into the camera coordinate system and the coordinates or positions of the feature points in the real-time collected facial image of the person.

For example, the optimization function can be expressed as the following formula:

Among them, B is the virtual model after driving, R is the rotation matrix, and t is the translation distance. The translation distance t can be determined by the distance between the feature point and the key point.

The expression coefficient of the expression target body is W = [w ₁ ,w ₂ ,…,w _m ], and _Wi is the expression coefficient of the i-th expression target body.

S305 , driving the virtual model based on the expression coefficient of each of the expression targets, wherein the driven virtual model is dynamically displayed.

Specifically, the virtual model after driving is expressed by the formula

Calculated, where Base is the virtual model, _Bi is the i-th expression target body, _Wi is the expression coefficient of the i-th expression target body, and B is the virtual model after driving.

That is to say, the driven virtual model can imitate the facial expressions of the characters in the video stream, and can be displayed in real time in the electronic device by imitating the facial expressions of the characters in the video stream, such as dynamic display, so as to realize human-computer interaction with the user. The video stream can contain multiple 2D pictures, and the video stream can be pre-stored in the electronic device. When performing human-computer interaction, by obtaining the interaction information input by the user, according to the interaction information, the target video stream is retrieved from the multiple video streams, and according to the target video stream, the facial features of the characters in the multiple pictures in the target video stream are obtained, and based on each of the facial features of the characters and the virtual model, the expression coefficients of the multiple expression target bodies in the virtual model are obtained, and based on the expression coefficients of each of the expression target bodies, the virtual model is driven, wherein the driven virtual model can be dynamically displayed in the electronic device, so as to imitate the facial expressions of the characters in the target video stream and be displayed in real time on the electronic device, thereby realizing human-computer interaction with the user.

It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

Exemplary virtual model construction device

As shown in FIG3 , an embodiment of the present application provides a virtual model construction device 100, the device 100 comprising:

A model acquisition module 110 is used to acquire a three-dimensional model of a target person;

A processing module 120 is used to perform parameterization processing on the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model;

A transformation module 130, configured to transform each point in the standard model based on the target transformation matrix to obtain a transformed standard model;

The editing module 140 is used to edit the transformed standard model to obtain a virtual model, wherein the virtual model includes a plurality of expression targets.

In one possible implementation, the processing module includes:

Obtaining a first error term according to the constraint of each adjacent point in the three-dimensional model;

Obtaining a second error term according to the constraints of the transformation matrix of each point in the standard model;

Obtaining a third error term according to constraints on corresponding points in the standard model and the three-dimensional model;

A target transformation matrix is determined according to a weighted minimum value of the first error term, the second error term, and the third error term.

In one possible implementation, the first error term is given by the formula

Calculated, where E _S is the first error term, _Ti and T _j are transformation matrices,

The second error term is given by the formula

Calculated, where E _I is the second error term,

The third error term is given by the formula

Calculated, where _EC is the third error term, _Vi is the coordinate of the i-th point in the standard model after transformation by the transformation matrix, Ci is the coordinate of the i-th point in the three-dimensional model corresponding to the point Vi in the standard model,

The weight and minimum value are given by the formula

Calculated, where W _S , W _I , and W _C are weight values.

In one possible implementation, the transformation module includes:

In response to the user's operation of editing the expression target body, a plurality of the expression target bodies are constructed in the transformed standard model to obtain a virtual model.

It can be understood that the virtual model construction device provided in the embodiment shown in Figure 3 can be used to implement the technical solution of the method embodiment shown in Figure 1 of the present application, and its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

Example virtual model interactive device

As shown in FIG. 4 , another embodiment of the present application provides a virtual model interaction device 200, the device 200 comprising:

The interaction information acquisition module 210 is used to acquire the interaction information input by the user;

A video stream acquisition module 220, configured to acquire a video stream based on the interaction information, wherein the video stream includes a plurality of pictures;

An extraction module 230 is used to extract facial features of a person from each of the pictures;

An expression coefficient acquisition module 240 is used to acquire expression coefficients of a plurality of expression targets in the virtual model based on the facial features of the character and the virtual model, wherein the virtual model is obtained by the method provided in the method embodiment shown in FIG. 1 ;

The driving module 250 is used to drive the virtual model based on the expression coefficient of each expression target body, wherein the driven virtual model is dynamically displayed.

In one possible implementation, the extraction module includes:

Acquire the collected facial image of the person from each of the pictures;

Feature extraction is performed on the facial image of the person to obtain facial features of the person.

In one possible implementation, the extraction module includes:

Acquire the collected voice signal from each of the pictures;

The mouth shape of each word in the speech signal is recognized to obtain the facial features of the person.

In one possible implementation, there is a corresponding relationship between the feature points in the facial features of the character and the key points in the virtual model, and the expression coefficient acquisition module includes:

The expression coefficient of each expression target body is determined according to the positional relationship between the feature points and the key points.

In one possible implementation, the virtual model after driving is represented by the formula

It is calculated, where Base is the virtual model, _Bi is the i-th expression target body, _Wi is the expression coefficient of the i-th expression target body, and B is the virtual model after driving.

It can be understood that the virtual model interaction device 200 provided in the embodiment shown in Figure 4 can be used to execute the technical solution of the method embodiment shown in Figure 2 of the present application, and its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

Exemplary Electronic Devices

FIG5 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. As shown in FIG5 , the electronic device may include: one or more processors; a memory; and one or more computer programs.

Among them, the above-mentioned electronic devices can be mobile phones, computers, servers, mobile terminals (mobile phones), intelligent robots, computers, smart screens, drones, intelligent connected vehicles (Intelligent Connected Vehicle; hereinafter referred to as: ICV), smart/intelligent cars or vehicle-mounted devices and other devices.

The one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the device, the device performs the following steps:

Obtain a three-dimensional model of the target person;

Parameterizing the three-dimensional model using a standard model to obtain a target transformation matrix for each point in the standard model;

Transforming each point in the standard model based on the target transformation matrix to obtain a transformed standard model;

The transformed standard model is edited to obtain a virtual model, wherein the virtual model includes a plurality of expression target bodies.

In one possible implementation, when the above instruction is executed by the above device, the above device performs parameterization processing on the three-dimensional model using the standard model to obtain a target transformation matrix for each point in the standard model, including:

Obtaining a first error term according to the constraint of each adjacent point in the three-dimensional model;

Obtaining a second error term according to the constraints of the transformation matrix of each point in the standard model;

Obtaining a third error term according to constraints on corresponding points in the standard model and the three-dimensional model;

A target transformation matrix is determined according to a weighted minimum value of the first error term, the second error term, and the third error term.

In one possible implementation, the first error term is given by the formula

Calculated, where E _S is the first error term, _Ti and T _j are transformation matrices,

The second error term is given by the formula

Calculated, where E _I is the second error term,

The third error term is given by the formula

Calculated, where _EC is the third error term, _Vi is the coordinate of the i-th point in the standard model after transformation by the transformation matrix, Ci is the coordinate of the i-th point in the three-dimensional model corresponding to the point Vi in the standard model,

The weight and minimum value are given by the formula

Calculated, where W _S , W _I , and W _C are weight values.

In one possible implementation, when the instruction is executed by the device, the device edits the transformed standard model to obtain a virtual model, including:

In response to the user's operation of editing the expression target body, a plurality of the expression target bodies are constructed in the transformed standard model to obtain a virtual model.

In one possible implementation manner, when the above instruction is executed by the above device, the above device performs the following steps:

Get interactive information input by the user;

Based on the interaction information, a video stream is acquired, wherein the video stream includes a plurality of pictures;

Extracting facial features of the person from each of the pictures;

Based on the facial features of the character and the virtual model, acquiring expression coefficients of multiple expression targets in the virtual model, wherein the virtual model is obtained by the method provided in the method embodiment shown in FIG1 ;

Based on the expression coefficient of each of the expression targets, the virtual model is driven, wherein the driven virtual model is dynamically displayed.

In one possible implementation, when the above instruction is executed by the above device, the above device executes the step of extracting facial features of a person from each of the pictures, including:

Acquire the collected facial image of the person from each of the pictures;

Feature extraction is performed on the facial image of the person to obtain facial features of the person.

In one possible implementation, when the above instruction is executed by the above device, the above device executes the step of extracting facial features of a person from each of the pictures, including:

Acquire the collected voice signal from each of the pictures;

The mouth shape of each word in the speech signal is recognized to obtain the facial features of the person.

In one possible implementation, there is a correspondence between feature points in the facial features of the person and key points in the virtual model. When the above instruction is executed by the above device, the above device executes the acquisition of expression coefficients of multiple expression targets in the virtual model based on the facial features of the person and the virtual model, including:

The expression coefficient of each expression target body is determined according to the positional relationship between the feature points and the key points.

In one possible implementation, the virtual model after driving is represented by the formula

It is calculated, where Base is the virtual model, _Bi is the i-th expression target body, _Wi is the expression coefficient of the i-th expression target body, and B is the virtual model after driving.

The electronic device shown in FIG5 can be a terminal device or a circuit device built into the terminal device. The device can be used to execute the functions/steps in the construction method provided in the embodiment shown in FIG1 of the present application, or the device can be used to execute the functions/steps in the interaction method provided in the embodiment shown in FIG2 of the present application.

As shown in FIG5 , the electronic device 900 includes a processor 910 and a memory 920. The processor 910 and the memory 920 can communicate with each other through an internal connection path to transmit control and/or data signals. The memory 920 is used to store computer programs, and the processor 910 is used to call and run the computer program from the memory 920.

The above-mentioned memory 920 can be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or it can be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or it can also be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

The processor 910 and the memory 920 may be combined into a processing device, or more commonly, they are independent components, and the processor 910 is used to execute the program code stored in the memory 920 to implement the above functions. In specific implementation, the memory 920 may also be integrated into the processor 910, or may be independent of the processor 910.

It should be understood that the electronic device 900 shown in FIG. 5 can implement each process of the method provided in the embodiment shown in FIG. 1 or FIG. 2 of the present application. The operations and/or functions of each module in the electronic device 900 are respectively to implement the corresponding processes in the above method embodiments. For details, please refer to the description in the method embodiment shown in FIG. 1 or FIG. 2 of the present application. To avoid repetition, the detailed description is appropriately omitted here.

In addition, in order to make the functions of the electronic device 900 more complete, the electronic device 900 may also include one or more of a camera 930, a power supply 940, an input unit 950, etc.

Optionally, the power supply 950 is used to provide power to various devices or circuits in the electronic device.

It should be understood that the processor 910 in the electronic device 900 shown in Figure 5 can be a system on chip SOC, which can include a central processing unit (Central Processing Unit; hereinafter referred to as: CPU), and can further include other types of processors, such as: a graphics processor (Graphics Processing Unit; hereinafter referred to as: GPU), etc.

In summary, the various processors or processing units within the processor 910 can work together to implement the previous method flow, and the corresponding software programs of the various processors or processing units can be stored in the memory 920.

The present application also provides an electronic device, which includes a storage medium and a central processing unit. The storage medium may be a non-volatile storage medium, in which a computer executable program is stored. The central processing unit is connected to the non-volatile storage medium and executes the computer executable program to implement the method provided in the embodiment shown in Figure 1 or Figure 2 of the present application.

In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the embodiment shown in Figure 1 or Figure 2 of the present application.

An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided by the embodiment shown in Figure 1 or Figure 2 of the present application.

Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The advantages of the present invention have been fully and effectively realized. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims (8)

1. A method of constructing a virtual model, comprising:

acquiring a three-dimensional model of a target person;

Parameterizing the three-dimensional model by using a standard model to obtain a target transformation matrix of each point in the standard model;

performing transformation processing on each point in the standard model based on the target transformation matrix to obtain a transformed standard model;

Editing the transformed standard model to obtain a virtual model, wherein the virtual model comprises a plurality of expression targets;

the parameterizing the three-dimensional model by using a standard model to obtain a target transformation matrix of each point in the standard model comprises the following steps:

Obtaining a first error term according to the constraint of each adjacent point in the three-dimensional model;

Obtaining a second error term according to the constraint of the transformation matrix of each point in the standard model;

obtaining a third error term according to the constraint of the corresponding points in the standard model and the three-dimensional model;

Determining a target transformation matrix according to the weighted sum minimum value of the first error term, the second error term and the third error term;

the first error term is represented by the formula

Calculated, where E _S is the first error term, T _i and T _j are the transformation matrices,

The second error term is represented by the formula

Calculated, where E _I is the second error term,

The third error term is represented by the formula

The calculation is carried out, wherein E _C is a third error term, V _i is the coordinate of the ith point in the standard model after transformation matrix transformation, C _i is the coordinate of the ith point in the three-dimensional model corresponding to the V _i point in the standard model,

The weights and minima are calculated by the formula

And calculating, wherein W _S、W_I、W_C is a weight value.

2. The method of claim 1, wherein editing the transformed standard model to obtain a virtual model comprises:

And responding to the operation of editing the expression target body by a user, and constructing a plurality of expression target bodies in the transformed standard model to obtain a virtual model.

3. A method of interacting with a virtual model, comprising:

acquiring interaction information input by a user;

acquiring a video stream based on the interaction information, wherein the video stream comprises a plurality of pictures;

Extracting facial features of people from each picture;

acquiring expression coefficients of a plurality of expression targets in a virtual model based on the character facial features and the virtual model, the virtual model being obtained by the method according to any one of claims 1 to 2;

And driving the virtual model based on the expression coefficient of each expression target, wherein the driven virtual model is dynamically displayed.

4. A method according to claim 3, wherein said extracting facial features of the person from each of said frames comprises:

Acquiring an acquired face image of a person from each picture;

And extracting the characteristics of the face image of the person to obtain the facial characteristics of the person.

5. A method according to claim 3, wherein said extracting facial features of the person from each of said frames comprises:

Acquiring an acquired voice signal from each picture;

and recognizing the mouth shape of each word in the voice signal to obtain the facial features of the person.

6. The method according to any one of claims 4 to 5, wherein the driven virtual model is represented by the formulaAnd calculating, wherein Base is a virtual model, B _i is an ith expression target body, W _i is an expression coefficient of the ith expression target body, and B is a driven virtual model.

7. A virtual model constructed by the method of any one of claims 1 to 2 for display in an electronic device.

8. An electronic device, comprising:

One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions, which when executed by the device, cause the device to perform the method of any of claims 1-6.