CN116617663B - Action instruction generation method and device, storage medium and electronic equipment - Google Patents
Action instruction generation method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN116617663B CN116617663B CN202210118989.0A CN202210118989A CN116617663B CN 116617663 B CN116617663 B CN 116617663B CN 202210118989 A CN202210118989 A CN 202210118989A CN 116617663 B CN116617663 B CN 116617663B
- Authority
- CN
- China
- Prior art keywords
- target
- current
- track
- information
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000009471 action Effects 0.000 title claims abstract description 240
- 238000000034 method Methods 0.000 title claims abstract description 92
- 238000003860 storage Methods 0.000 title claims abstract description 32
- 230000033001 locomotion Effects 0.000 claims abstract description 363
- 239000013598 vector Substances 0.000 claims description 337
- 230000005021 gait Effects 0.000 claims description 207
- 230000004927 fusion Effects 0.000 claims description 38
- 238000004590 computer program Methods 0.000 claims description 22
- 238000012937 correction Methods 0.000 claims description 19
- 238000003062 neural network model Methods 0.000 claims description 18
- 230000001960 triggered effect Effects 0.000 claims description 6
- 239000000463 material Substances 0.000 abstract description 16
- 238000013473 artificial intelligence Methods 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 5
- 230000000875 corresponding effect Effects 0.000 description 185
- 210000003414 extremity Anatomy 0.000 description 48
- 210000001503 joint Anatomy 0.000 description 27
- 210000002414 leg Anatomy 0.000 description 22
- 238000010586 diagram Methods 0.000 description 14
- 230000001276 controlling effect Effects 0.000 description 13
- 238000013528 artificial neural network Methods 0.000 description 11
- 210000001364 upper extremity Anatomy 0.000 description 11
- 238000012545 processing Methods 0.000 description 10
- 241001465754 Metazoa Species 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 210000000544 articulatio talocruralis Anatomy 0.000 description 6
- 238000004088 simulation Methods 0.000 description 6
- 238000010171 animal model Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 230000009191 jumping Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 210000002683 foot Anatomy 0.000 description 4
- 210000001015 abdomen Anatomy 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013136 deep learning model Methods 0.000 description 3
- 210000001624 hip Anatomy 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 230000007704 transition Effects 0.000 description 3
- 206010034701 Peroneal nerve palsy Diseases 0.000 description 2
- 238000013016 damping Methods 0.000 description 2
- 239000002360 explosive Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 210000003141 lower extremity Anatomy 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009192 sprinting Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000004804 winding Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000037237 body shape Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- -1 for example Substances 0.000 description 1
- 210000004394 hip joint Anatomy 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005304 joining Methods 0.000 description 1
- 210000000629 knee joint Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/55—Controlling game characters or game objects based on the game progress
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Human Computer Interaction (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application discloses a method and a device for generating an action instruction, a storage medium and electronic equipment, which can be applied to the field of artificial intelligence. Wherein the method comprises the following steps: acquiring action state information of limbs of a controlled object on a corresponding target point at a current moment, wherein the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment; performing motion state prediction based on the motion state information to obtain motion state reference information of the target point at the next moment; updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment; and generating a predicted action instruction of the controlled object at the next moment based on the action state estimated information. The method solves the problem that the storage space and the running space are occupied greatly due to excessive required action materials in the action instruction generation method in the related technology.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method and apparatus for generating an action instruction, a storage medium, and an electronic device.
Background
Currently, the controlled object may be controlled by the action instruction to perform a corresponding action, for example, in a game application or other scenes, an object action may be generated for the controlled object, and the generated object action may be a series of actions performed by the controlled object. The mode of generating the action instruction for the controlled object in the related art is as follows: and reading an action material library from the memory, and then controlling the controlled object to execute corresponding actions according to the closest action instruction matched in real time from the action material library according to the object state of the controlled object.
However, in the above-described action instruction generation method, in order to increase the richness of the execution action, it is necessary to increase the action materials in the action material library. As the linearity of the motion materials increases, the memory space and the computation amount required by the motion materials also increase linearly, and the increase of the motion materials occupies more memory space and operation memory.
Therefore, the action instruction generating method in the related art has the problem that the storage space and the running space are occupied greatly due to the excessive action materials.
Disclosure of Invention
The embodiment of the application provides a method and a device for generating action instructions, a storage medium and electronic equipment, which at least solve the problem that the storage space and the running space are occupied greatly due to excessive required action materials in the action instruction generating method in the related technology.
According to an aspect of an embodiment of the present application, there is provided an action instruction generation method including: acquiring action state information of limbs of a controlled object on a corresponding target point at a current moment, wherein the action state information comprises track sequence information of the target point at a group of historical moment, current moment and a group of future moment; performing motion state prediction based on the motion state information to obtain motion state reference information of the target point at the next moment; updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment; and generating a predicted action instruction of the controlled object at the next moment based on the action state estimated information.
According to another aspect of the embodiment of the present application, there is also provided an action instruction generating apparatus, including: the system comprises an acquisition unit, a control unit and a control unit, wherein the acquisition unit is used for acquiring action state information of limbs of a controlled object on a corresponding target point at the current moment, and the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment; the prediction unit is used for predicting the action state based on the action state information to obtain action state reference information of the target point at the next moment; the updating unit is used for updating the action state information according to the action state reference information to obtain action state estimated information of the target point at the next moment; and the generating unit is used for generating a predicted action instruction of the controlled object at the next moment based on the action state estimated information.
According to a further aspect of embodiments of the present application, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to perform the above-described action instruction generation method when run.
According to yet another aspect of embodiments of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device performs the action instruction generating method as above.
According to still another aspect of the embodiments of the present application, there is also provided an electronic device including a memory in which a computer program is stored, and a processor configured to execute the above-described action instruction generation method by the computer program.
In the embodiment of the application, when the controlled object is instructed to execute the action through the action instruction, the action state information of the limb pair of the controlled object corresponding to the target point at the current moment can be obtained, the limb pair and the target point are in one-to-one correspondence, the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment, and by setting the correspondence between the limb pair and the target point, effective control of each limb of the controlled object can be ensured; according to the method, the system and the device, the motion state information of a target point at the next moment is obtained, the motion state information is updated according to the motion state information, the motion state estimated information of the target point at the next moment is obtained, a predicted motion instruction of a controlled object at the next moment is generated according to the motion state estimated information, motion can be generated for the controlled object without matching motion materials in a motion material library in the mode of data-driven motion generation, occupation of storage space and operation space can be reduced, and the generated motion of the controlled object is more vivid and has better expressive force according to the mode of motion generation of the corresponding relation between the limb pair and the target point, so that the technical effects of reducing occupation of the storage space and the operation space and improving the simulation degree of motion generation are achieved, and the problem that the occupation of the storage space and the operation space is large due to the fact that the needed motion materials are too much in the motion instruction generation method in the related technology is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
FIG. 1 is a schematic illustration of an application environment of an alternative method of action instruction generation in accordance with an embodiment of the present application;
FIG. 2 is a schematic illustration of an application environment of another alternative method of action instruction generation according to an embodiment of the present application;
FIG. 3 is a flow chart of an alternative method of generating action instructions according to an embodiment of the application;
FIG. 4 is a schematic diagram of an alternative data-driven action animation real-time generation in accordance with an embodiment of the present application;
FIG. 5 is a flow diagram of an alternative model training according to an embodiment of the application;
FIG. 6 is a flow diagram of another alternative model training in accordance with an embodiment of the present application;
FIG. 7 is a schematic diagram of an alternative action generation model according to an embodiment of the application;
FIG. 8 is a schematic diagram of an alternative keypoint in accordance with embodiments of the application;
FIG. 9 is a schematic diagram of an alternative historical track sequence update according to an embodiment of the application;
FIG. 10 is a schematic diagram of an alternative track sequence update in accordance with an embodiment of the application;
FIG. 11 is a schematic diagram of animation generation under alternative user input control in accordance with an embodiment of the application;
FIG. 12 is a schematic diagram of an alternative method of action instruction generation according to an embodiment of the present application;
FIG. 13 is a flow chart of another alternative method of generating action instructions according to an embodiment of the application;
FIG. 14 is a block diagram of an alternative action instruction generating device according to an embodiment of the present application;
FIG. 15 is a schematic diagram of an alternative electronic device in accordance with an embodiment of the application;
FIG. 16 is a block diagram of the architecture of a computer system of an alternative electronic device in accordance with an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of an embodiment of the present application, there is provided an action instruction generation method. Alternatively, as an alternative embodiment, the above-described method of generating an action instruction may be applied, but not limited to, in the environment shown in fig. 1 and 2. Among others, this may include, but is not limited to: a terminal device 102, the terminal device 102 may include, but is not limited to, a memory 104, a processor 106, and a display 108; a network 110; a server 112.
Alternatively, in the present embodiment, the terminal device 102 may include, but is not limited to, at least one of the following: a Mobile phone (e.g., an Android Mobile phone, iOS Mobile phone, etc.), a notebook computer, a tablet computer, a palm computer, a MID (Mobile INTERNET DEVICES, mobile internet device), a PAD, a desktop computer, a vehicle-mounted device, a robot (i.e., a controlled robot), etc. The network may include, but is not limited to: a wired network, a wireless network, wherein the wired network comprises: local area networks, metropolitan area networks, and wide area networks, the wireless network comprising: bluetooth, WIFI (WIRELESS FIDELITY ) and other networks that enable wireless communications. The server 112 may be a single server, a server cluster including a plurality of servers, or a cloud server. The above is merely an example, and is not limited in any way in the present embodiment.
Illustratively, as shown in fig. 1, taking the terminal device 102 as an example of a device capable of executing a predicted action instruction or displaying a game screen, the procedure of the action instruction generation method described above may include the steps of:
In step S102, the server 112 acquires motion state information of a virtual character (an example of a controlled object) corresponding to the current frame, generates a predicted motion instruction of the virtual character in the next frame according to the motion state information, and renders a game screen of the next frame based on the generated motion prediction instruction.
For a scenario in which a virtual character is controlled to perform a game action in a game application, the server 112 may obtain action state information corresponding to a current frame of the virtual character, where the virtual character may include a set of limb pairs, where the action state information is action state information of a root (i.e., a root) point (an example of a target point) corresponding to each limb pair in the set of limb pairs in the current frame, and may include track sequence information of the root point in a set of historical moments, a current moment, and a set of future moments.
Based on the motion state information, the server 112 may perform motion state prediction of the virtual character to obtain motion state reference information of the root point in the next frame. According to the motion state reference information, the server 112 may update the motion state information to obtain motion state estimated information of the root point in the next frame. Based on the motion state estimation information, the server 112 may generate a predicted motion instruction of the virtual character in the next frame (may correspond to the motion of the virtual character in the next frame), and further render a game screen in the next frame.
In step S104, the server 112 may transmit the game screen of the next frame to the terminal device 102 through the network 110.
In step S106, the terminal device 102 displays a game screen of the next frame in its game interface.
The terminal device 102 may receive the game screen of the next frame transmitted from the server 112 and update the game screen displayed in its game interface from the game screen of the current frame to the game screen of the next frame when the game screen is refreshed.
In addition to the example shown in fig. 1, the above steps may be independently performed by the terminal device 102, that is, the terminal device 102 performs the steps of updating the status information of the virtual character, generating the predicted action instruction, generating the game screen, and the like, thereby reducing the processing pressure of the server.
Illustratively, as shown in fig. 2, taking the terminal device 102 as an example of a controlled robot capable of executing a predicted action instruction, the process of the action instruction generating method may include the following steps:
in step S202, the server 112 acquires the operation state information of the controlled robot (an example of a controlled object) corresponding to the current time, and generates a predicted operation command of the passive robot at the next time based on the operation state information.
For a scenario of controlling the controlled robot to perform an actual action, the server 112 may acquire action state information of the controlled robot corresponding to the current moment, where the controlled robot may include a set of limb pairs, where the action state information is action state information of a root point corresponding to each limb pair in the set of limb pairs at the current moment, and may include track sequence information of the root point at a set of historical moments, the current moment, and a set of future moments.
Based on the motion state information, the server 112 may predict the motion state of the controlled robot, and obtain motion state reference information of the root point at the next moment. According to the motion state reference information, the server 112 may update the motion state information to obtain motion state estimated information of the root point at the next moment. Based on the motion state estimation information, the server 112 may generate a predicted motion instruction of the controlled robot at the next time (may correspond to the motion of the controlled robot at the next time).
In step S104, the server 112 may transmit the predicted action instruction of the next time to the terminal device 102 through the network 110.
In step S106, the terminal device 102 controls the controlled robot to execute the predicted action instruction at the next time.
The terminal device 102 (i.e., the controlled robot) may receive the predicted action instruction of the next time transmitted from the server 112 and execute the predicted action instruction at the next time.
In addition to the example shown in fig. 2, the above steps may be independently performed by the terminal device 102, that is, the terminal device 102 performs the steps of updating the state information of the controlled robot, generating the predicted action instruction, performing the predicted action instruction, and the like, thereby reducing the processing pressure of the server.
Alternatively, taking the above-mentioned server 112 as an example of the action instruction generating method in this embodiment as an alternative implementation manner, fig. 3 is a schematic flow chart of an alternative action instruction generating method according to an embodiment of the present application, and as shown in fig. 3, the flow of the action instruction generating method may include the following steps:
Step S302, motion state information of limbs of a controlled object at the current moment of a corresponding target point is obtained, wherein the motion state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment.
The action instruction generation method in the embodiment of the application can be applied to a scene that generates an action instruction for a controlled object through data driving and controls the controlled object to execute corresponding actions according to the generated action instruction. Optionally, the controlled object may be a virtual character, or may be a controlled robot. The above-described action instruction generation method may be applied to the field of artificial intelligence (ARTIFICIAL INTELLIGENCE, abbreviated as AI), and may be used to produce intelligent machines (e.g., the controlled robots) that can react in a manner similar to human intelligence.
It should be noted that artificial intelligence is a theory, a method, a technique, and an application system that simulates, extends, and expands human intelligence using a digital computer or a machine controlled by a digital computer, senses an environment, acquires knowledge, and obtains an optimal result using the knowledge. That is, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence, giving the machine the functions of sensing, reasoning and decision.
When the controlled object is a virtual character, executing the corresponding action according to the generated action instruction can be used for generating the action animation of the virtual character. The motion animation may be a motion animation in an application having a motion animation generation requirement, such as a game application, a video application, or an animation generation, and may be a motion animation generated in real time, a motion animation generated in non-real time, or a motion animation in another scene.
When the controlled object is a controlled robot, the control of executing the corresponding action according to the generated action instruction may be controlling the controlled robot to execute actions such as jumping, sitting down, lying down, running, standing, etc. and executing switching between different actions according to the generated action instruction. The controlled robot may be a robot that performs a specific task, for example, a companion robot, a robot in movie production, or another type of intelligent robot, which is not limited in this embodiment.
In the present embodiment, the controlled object may be an object having a plurality of legs or a plurality of feet (i.e., a multi-foot object), for example, a four-foot virtual animal, a four-foot robot, or the like. In this embodiment, morphological characteristics and the like of the controlled object are not limited. Alternatively, in some examples, the controlled object is exemplified by a quadruped or quadruped robot in a gaming application.
In order to reduce the excessive occupation of storage space and running memory caused by the growth of action materials, a data-driven action instruction generation mode can be adopted. The scheme of generating the data-driven action instruction may be to input the current action state of the controlled object, or the current action state and the control instruction input by the user object or other control objects into a deep learning model, a neural network model or other prediction models to predict the action state, so as to predict the action state at the next moment. Further, a predicted action instruction of the controlled object may be generated based on the predicted action state.
For example, taking a virtual character as an example in connection with fig. 4, given a current character state (i.e., current state information of the virtual character corresponding to the current frame, an example of the above-mentioned motion state information) which may include a position and a speed of a joint of the character, a position and a direction of a target track, surrounding topography data, etc., while a control instruction is input by a user, a prediction model (a deep learning model) may predict a character state of the virtual character of the next frame according to the current state and the input control instruction, and update the character state of the virtual character in the game in real time.
The prediction model may be obtained by performing model training using collected training data, and the training data used for the prediction model of the different types of virtual characters may be different. For example, as shown in fig. 5, for a scene in which data-driven bipedal characters (e.g., bipedal characters) are animated, a person of interest may wear a garment with a tailored sensor to perform an action requiring acquisition, e.g., walking, running, jumping, etc. For another example, as shown in fig. 6, for a scene in which data-driven quadruped animals are animated, a garment with a special sensor may be worn by an actual animal or the like to perform an action to be acquired. Because factors such as heights, body shapes, joint numbers and the like of different people and different animals are different from roles in the virtual world, the acquired action sequences need to be mapped into virtual characters, the mapped action sequences can be used for model training, and the model can be deployed after model training is completed.
For bipedal character models (e.g., bipedal upright walking character models, which may be bipedal character models), the data-driven motion animation generation methods employed mainly include the following two types: PFNN (Phase-Functioned Neural Network, phase neural network) and NSM (Neural STATE MACHINE, finite state machine). PFNN proposes a neural network based on a phase function, which can dynamically change the network weight, learn how to move from a large amount of highly-dimensional data with high coupling between environment geometry and character actions, and further stabilize the action animation of the generated virtual character on line in real time. NSM is a direct replacement of the PFNN phase function with another neural network to fit a more complex mapping between phase, gait information and expert weights. NSM can interact with the target object accurately through a bidirectional control mechanism, and the action range capable of generating the animation is greatly expanded.
However, the two networks PFNN and MSN are mainly applied to the motion animation generation of the bipedal roles, and if the two networks are applied to the motion animation generation of the quadruped animal model, the phenomenon that the model wholly floats forwards but the four legs do not swing easily occurs. Therefore, PFNN and NSM are generally used only for generating motion animation of a virtual character walking upright on both feet.
For quadruped animals MANN (Mode-Adaptive Neural Network, mode-adaptive network) can be used to generate motion animation. Compared to NSM, MANN changes the gate network input: the gate network input of NSM is the Cronecker product of phase and gait; MANN is to drive character motion performance in a game by correlation of velocity vectors at four leg ends and animal motion, and the gate network input has no phase, only gait and velocity vectors at four leg ends.
However, the motion generation model based on MANN has only one root, if the root is set between two front legs of the quadruped animal model, the two front legs can alternately advance, and the two rear legs can appear to be dragged in the air; on the contrary, if the root is arranged between two rear legs of the quadruped animal model, the two rear legs are vivid, the two front legs are stiff and do not move, and the feet float in the air. Moreover, it is sufficient to use a single root locus to characterize the motion timing information of the bipedal upright walking character model, but it is difficult to vividly express, distinguish the quadruped from the motions unique to bipedal upright walking, such as tail flick, heel strike, etc., by characterizing the quadruped animal model advancing on the ground.
Additionally, motion animation may be generated using an LMP (Local Motion Phase ) based motion generation model. The LMP is to assign a phase to each of the four legs, and manually mark the action of each leg in the action capturing data by a large amount of manpower, and the needed manpower resource is large because of manual inspection.
It should be noted that, for a scenario in which the controlled object is a controlled robot, the motion generating model (for example, PFNN, MSN, MANN and LMP) may also be used to predict the motion state of the controlled robot at the next time.
For a controlled object (for example, a quadruped model, which is also applicable to other types of controlled objects), an intelligent action generation mode is provided in the embodiment of the application, and the intelligent action generation mode can be applied to an action generation model shown in fig. 7. The action generation model includes: and a motion prediction network, which inputs a current motion state (for example, motion state information may include a track position, a direction, a speed, a joint position, a direction, a speed, a gait type, a phase, etc.) of the controlled object, and outputs a predicted motion state (for example, motion state estimation information) at the next moment. Alternatively, the subset of the current state may be the gate network input, the output of the gate network is the weight of the hybrid expert model, and the network weights of the motion prediction network may be obtained using a weighted average of these weights.
The network structure of the action generation model described above is similar to PFNN, NSM, MANN and LMP described above. Wherein PFNN has no gate network, it uses a fixed phase function to generate the weights of the 4-dimensional hybrid expert model. NSM differs from MANN only in the selection of the gate network input state: NSM selects information such as trajectory, gait, phase, etc. as gate network input, and MANN selects trajectory speed, toe end speed, gait as gate network input. LMP (if used to treat quadruped animals) tags all four legs with phases and then includes information of all four phases in the input of the gate network.
The deployment flow of the action generation model comprises the following steps: data acquisition, data annotation, data preprocessing, model training and model deployment. Compared with PFNN, NSM, MANN and LMP described above, the motion prediction network of the motion generation model in this embodiment inputs motion state information of the limb of the controlled object at the current time point of the corresponding target point, where the motion state information includes track sequence information of the target point at a set of historical time points, the current time point and a set of future time points. Here, the number of limbs of the controlled object may be an even number, and the even number of limbs may form a group of limb pairs (including at least one limb pair or including at least two limb pairs), and each limb pair may include at least two limbs, for example, two front legs, two rear legs, and the like. For example, a virtual character or passive robot may have at least four legs and an even number of legs. The limb pairs and the target points may be in one-to-one correspondence, i.e., each limb pair corresponds to one target point, which may be considered as the root point of the corresponding limb pair.
Taking a quadruped animal model as an example, a track in the input of the motion prediction network is changed into a double track, the double track can be a track of track points of a track motion generation model arranged between two front legs and between two rear legs, motion time sequence information of the virtual character is represented by using the double track, unique motions of the virtual character different from the motions of bipedal vertical walking, such as tail flick, back flip and the like, can be vividly represented, so that the multi-foot drop points of the virtual character have expressive force and explosive force of waist and abdomen of the virtual character can be more vividly represented, and the simulation degree of the virtual character is improved.
When generating the action instruction based on the action generating model, the server can acquire the action state information, namely, the action state information of the limb pair of the controlled object corresponding to the target point at the current moment. The number of the limb pairs can be one or more, and the limb pairs and the target points are in one-to-one correspondence. In this case, the server may acquire the motion state information of the target point corresponding to each limb pair of the group of limb pairs of the controlled object at the current time.
The motion state information includes track sequence information of each target point at a set of historical time, current time and a set of future time, and the track sequence information may include values of a parameter corresponding to the target point at the set of historical time, current time and the set of future time. If there are multiple target points, the action state information may include values of a parameter corresponding to each target point at a set of historical time, current time and a set of future time. If the parameters corresponding to the target point are multiple, the action state information may include values of each parameter at a set of historical time, current time and a set of future time.
The set of historical time instants, the current time instant and the set of future time instants may be a plurality of consecutive time instants (here, consecutive time instants are consecutive time instants based on the output frame or the refresh frequency, i.e. a plurality of consecutive output frame time instants, a plurality of consecutive refresh time instants) or may be discrete time instants. The relative relationship between the number of times included in the set of historical times, the current time, and the set of future times, and the time, may be configured as desired, and is not limited in this embodiment.
For a scenario in which the controlled object is a virtual character, the virtual character may be a virtual character in the target application, for example, a quadruped in the target game application. The target game application may render the game frames at a game frame rate (e.g., 60 frames/s for the game, i.e., 60 frames rendered in one second), and for each frame in the game (i.e., EVERY TICK), may generate an action of the virtual character in each frame based on the state information of the virtual character, thereby resulting in an action animation of the virtual character.
For a scenario in which the controlled object is a controlled robot, the controlled robot may be a quadruped robot, and the controlled robot may refresh the motion state of the passive robot at a certain refresh frequency (for example, the refresh frequency is 60 times/s, that is, 60 times in one second), and for each refresh time, a predicted motion instruction of the controlled robot at the next time may be generated based on the state information of the controlled robot.
Step S304, the action state is predicted based on the action state information, and the action state reference information of the target point at the next moment is obtained.
After the action state information is obtained, the server can predict the action state based on the action state information, and the obtained prediction result is action state reference information of the target point at the next moment. The obtained motion state reference information may include track prediction information (corresponding to the track sequence information) of the target point at the next time, and the obtained track prediction information may be used to update track sequence information, where the update may be track sequence information corresponding to the current time and a set of future times in the track sequence information. In addition, the motion state reference information may further include other information, for example, joint point information of a preset joint point (i.e., a key point) of the controlled object, may include position information of the joint point, and may further include other reference information related to the motion state, which is not limited in this embodiment.
Alternatively, the motion state prediction may be performed using a target prediction neural network model. The target predictive neural network model may be a neural network model that is model trained and deployed by the model training approach described above. The target prediction neural network model may include an action prediction network, and the target prediction neural network model may first extract information of the action state information to obtain an input of the prediction network corresponding to the action prediction network, which may be an M-dimensional feature vector, for example, extract information from the action state information s i, and arrange the information into a format of an input x i of the action prediction network. Optionally, the target prediction neural network model may further include a gate network, and the target prediction neural network model may further predict a part of inputs in the network inputs, and determine the part of inputs as inputs of the gate network, where the inputs of the gate network may be N-dimensional feature vectors, and M is greater than or equal to N.
After the predicted network input is obtained, the predicted network input can be input into the motion prediction network to obtain motion state reference information output by the motion prediction network. When the target prediction neural network model includes a gate network, the input of the gate network may be first put into the gate network to obtain the output of the gate network, the output of the gate network is used as a weight to perform weighted average on the expert weights in the hybrid expert model to obtain the network weights of the motion prediction network, then the obtained network weights are loaded into the motion prediction network, and then the input x i of the well-sorted motion prediction network is input, and the output y i (i.e., the motion state reference information) is obtained through forward calculation of the motion prediction network.
Step S306, the motion state information is updated according to the motion state reference information, and motion state estimated information of the target point at the next moment is obtained.
After the motion state reference information is obtained, the server may update the motion state information by using the motion state reference information, for example, perform neural network output fusion to obtain motion state estimated information of the controlled object, where the motion state estimated information is initial state information of the target point at the next moment. When no control information for controlling the controlled object is detected, the motion state information of the target point at the next moment is motion state estimated information. When the control information for controlling the controlled object is detected, the action state information of the target point at the next moment is action state estimated information after being adjusted by combining the input control information.
The updating of the motion state information may include updating track sequence information, for example, updating track sequence information using track prediction information, and may include updating track sequence information corresponding to a current time and a set of future times in the track sequence information. The update may be performed by direct replacement of the corresponding position data, or may be performed by weighted summation, which is not limited in this embodiment. Alternatively, the trajectory prediction information may be converted into a representation in the world coordinate system before the action state information is updated.
If the set of historical time, the current time and the set of future time are discontinuous time, the action state information can be updated by combining a set of historical track sequence information, wherein the set of historical track sequence information comprises values of a parameter corresponding to the target point at a plurality of continuous time points. The updating of the action state information in combination with the set of historical track sequence information may be updating the track sequence information using the set of historical track sequence information, and may include updating track sequence information corresponding to the set of historical moments in the track sequence information.
Step S308, based on the motion state estimation information, a predicted motion instruction of the controlled object at the next moment is generated.
Based on the motion state estimation information, the server can generate a predicted motion of the controlled object at the next moment, so as to generate a predicted motion instruction of the controlled object at the next moment, wherein the predicted motion instruction is used for indicating the predicted motion of the controlled object at the next moment.
For the virtual character, when the picture rendering is needed, the server can extract information required by the picture rendering from the motion state estimated information and perform the picture rendering, so that the action of the controlled object on the next frame is generated, and the rendered picture is transmitted to the corresponding equipment for display.
For example, before each tick begins, the game program may extract corresponding information from the current state information (i.e., action state information) s i to render a game screen and play in the game interface.
For the controlled robot, the server may send the predicted action instruction directly to the controlled robot. After receiving the predicted action instruction, the controlled robot can execute the predicted action instruction at the next moment.
According to the embodiment of the application, the problem that the storage space and the running space are occupied greatly due to excessive required action materials in the action instruction generation method in the related art is solved based on the mode of action generation of the corresponding relation between the limb pair and the target point, the occupation of the storage space and the running space is reduced, and the simulation degree of action generation is improved.
As an alternative embodiment, the controlled object may be an object having at least four feet (or at least four legs) (e.g., a four-foot controlled object, etc.). In this case, the limb pair of the controlled object includes a first limb pair and a second limb pair, and the target point includes two root points: a target front point (i.e., a front root point) corresponding to the first limb pair, which is located at a point between two front legs in the controlled object, for example, the front root point may be a point located between two front legs on the ground right below the front end of the spine of the controlled object; the target rear point (i.e., rear root point) corresponding to the second limb pair is located at a point between two rear legs in the controlled object, for example, the rear root point may be a point located between two rear legs on the ground immediately below the rearmost end of the spine of the controlled object. The track sequence information may be determined based on the above-described target front point and target rear point, and may include a first set of track sequences (i.e., a set of front track sequences) corresponding to the target front point and a second set of track sequences (i.e., a set of rear track sequences) corresponding to the target rear point.
It should be noted that, unlike the single-root track used in other intelligent animation generation technologies, the present embodiment proposes a double-root track representation mode, and since the root point is generally located on the ground, the position, the direction, and the velocity vector of the root point may be only two-dimensional (for example, two dimensions of x and y).
Correspondingly, in this embodiment, obtaining the motion state information of the limb pair of the controlled object at the current time corresponding to the target point includes:
S11, a first group of track sequences of a target front point corresponding to a first limb pair of a controlled object at the current moment are obtained, wherein each track sequence in the first group of track sequences comprises values of a parameter corresponding to the target front point at a group of historical moment, the current moment and a group of future moments;
s12, a second group of track sequences of the target rear point corresponding to the second limb pair of the controlled object at the current moment is obtained, wherein each track sequence in the second group of track sequences comprises the values of one parameter corresponding to the target rear point at a group of historical moments, the current moment and a group of future moments.
In this embodiment, the server may obtain a first set of track sequences of the target front point at the current time, where each track sequence in the first set of track sequences includes values of a parameter corresponding to the target front point at a set of historical time, the current time, and a set of future time, and obtain a second set of track sequences of the target rear point at the current time, where each track sequence in the second set of track sequences includes values of a parameter corresponding to the target rear point at a set of historical time, the current time, and a set of future time.
Each track sequence in the first set of track sequences may correspond to a certain parameter of the target front point, which may include a value of the certain parameter corresponding to the target front point at a plurality of moments (which may be all or part of a set of historical moments, a current moment, and a set of future moments). The types of parameters corresponding to different track sequences in the first set of track sequences may be different, and the parameters may include, but are not limited to, at least one of the following: position, direction, velocity vector. Each track sequence in the second set of track sequences may correspond to a certain parameter of the target rear point, which may include values of a parameter corresponding to the target rear point at a plurality of moments. The type of parameters corresponding to different track sequences in the second set of track sequences may be different.
Here, the parameter corresponding to each track sequence may be a multidimensional vector, i.e., a parameter represented by a multidimensional vector, for example, a two-dimensional vector, a three-dimensional vector, a five-dimensional vector, etc., which may be used to characterize a track feature of the track sequence.
For example, the current state information of the current frame may include a set of front track sequences and a set of rear track sequences, each of which may correspond to one parameter of 13 root points, respectively, each of which is a 2-dimensional vector, i.e., each of which is a 13×2-dimensional vector. As shown in FIG. 8, the first six points are the real experienced states of the root in the past 1 second, 5/6 second, 4/6 second, 3/6 second, 2/6 second, 1/6 second; the seventh point represents the current root state; the last six points represent the states where the root point is expected to arrive at 1/6 second, 2/6 second, 3/6 second, 4/6 second, 5/6 second, 1 second in the future.
According to the embodiment provided by the application, the motion instruction is generated for the multi-foot controlled object based on the double tracks, so that the rationality of the generation of the motion instruction can be improved.
As an alternative embodiment, performing motion state prediction based on motion state information to obtain motion state reference information of a target point at a next moment, including:
S21, inputting the motion state information into a target prediction neural network model to obtain output motion state reference information, wherein the motion state reference information comprises a first group of sub-track sequences, and each sub-track sequence in the first group of sub-track sequences comprises a predicted value of a parameter corresponding to a target point at the current moment and a group of future moments.
The motion state reference information may comprise a first set of sub-track sequences, where each sub-track sequence in the first set of sub-track sequences comprises a predicted value of a parameter corresponding to the target point at the current time instant and at a set of future time instants. The first set of sub-track sequences are sub-track sequences in a coordinate system with the target point as a centre point, and therefore the first set of sub-track sequences can be converted into a representation in a world coordinate system before the motion state information is updated.
Optionally, the target point may include a front target point and a rear target point, and correspondingly, the first group of sub-track sequences may include a first group of front sub-track sequences corresponding to the front target point and a first group of rear sub-track sequences corresponding to the rear target point, where each sub-track sequence in the first group of front sub-track sequences includes a predicted value of a parameter corresponding to the front target point at a current time and a group of future times, and each sub-track sequence in the first group of rear sub-track sequences includes a predicted value of a parameter corresponding to the rear target point at the current time and a group of future times.
In this embodiment, the motion state prediction may be performed using the aforementioned target prediction neural network model. The server may input motion state information to the target predictive neural network model. And the target prediction neural network model analyzes the input action state reference information, so that the action state reference information output by the target prediction neural network model is obtained.
For example, the output y i of the motion-prediction network may include a plurality of pre-prediction track sequences (i.e., a first set of pre-sub-track sequences) and a plurality of post-prediction track sequences (i.e., a first set of post-sub-track sequences), each pre-prediction track sequence including: the local coordinates of one parameter of the front root point of the 7 key points at present and future are represented in the front root coordinate system, and each predicted track sequence comprises: the local coordinates of one parameter of the post-root point at the current and future 7 key points are represented in the post-root coordinate system.
The updating of the motion state information may include updating a first set of track sequences and a second set of track sequences, e.g., updating the first set of track sequences using a first set of front sub-track sequences and updating the second set of track sequences using a first set of rear sub-track sequences. The update may be performed by direct replacement of the corresponding position data, or may be performed by weighted summation, which is not limited in this embodiment.
According to the embodiment provided by the application, the motion state is predicted through the prediction neural network model, so that the predicted value of one parameter corresponding to the target point at the current moment and a group of future moments is obtained, and the accuracy and the rationality of motion state information prediction can be improved.
As an alternative embodiment, the track sequence information may comprise a set of track sequences, each track sequence of the set of track sequences comprising the values of a parameter corresponding to the target point at a set of historical moments, the current moment and a set of future moments. Alternatively, the target point may include the aforementioned pre-target point and post-target point, and the set of track sequences may include the aforementioned first set of track sequences and second set of track sequences.
Correspondingly, updating the motion state information according to the motion state reference information comprises the following steps:
S31, updating a group of track sequences according to a first group of sub-track sequences, wherein the predicted value of a parameter corresponding to a target point in each sub-track sequence in the first group of sub-track sequences at the current time and a group of future time is used for updating the value of a parameter corresponding to the target point in each track sequence in the group of track sequences at the current time and a group of future time.
In this embodiment, updating the action state information may include: a set of track sequences is updated using the first set of sub-track sequences. When a set of track sequences is updated, corresponding updating can be performed according to parameters corresponding to the target points. That is, for a parameter corresponding to the target point, a sub-track sequence corresponding to the parameter in the first group of sub-track sequences may be used to update a track sequence corresponding to the parameter in the group of track sequences, where the update may be performed according to a feature vector, that is, updating a corresponding position, that is, updating a value at a corresponding time.
Optionally, the predicted values of the parameters corresponding to the target points in each of the first set of sub-track sequences at the current time and the set of future times may be used to update the values of the parameters corresponding to the target points in each of the first set of track sequences at the current time and the set of future times.
When the target point includes a pre-target point and a post-target point, the first set of sub-track sequences may include the aforementioned first set of pre-sub-track sequences and first set of post-sub-track sequences. Updating the set of track sequences from the first set of sub-track sequences may include: updating the values of the parameters corresponding to the target front points in each track sequence in the first group of track sequences at the current time and at the set of future times according to the predicted values of the parameters corresponding to the target front points in each sub track sequence in the first group of front sub track sequences at the current time and at the set of future times, and updating the values of the parameters corresponding to the target rear points in each track sequence in the second group of track sequences at the current time and at the set of future times according to the predicted values of the parameters corresponding to the target rear points in each sub track sequence in the first group of rear sub track sequences at the current time and at the set of future times.
For example, one pre-prediction trajectory sequence in the output y i for the action prediction network may include: a local coordinate representation (which may be a 7 x 2 dimensional feature vector) of one parameter of the current and future 7 key point front root points in the front root coordinate system may first be converted to a representation in the world coordinate system and then used to update the corresponding parameters of the current and future 7 key point front root points in the track sequence corresponding to the input x i of the motion prediction network.
The sequence of predicted trajectories for one of the outputs y i of the motion prediction network may include: the local coordinate representation (which may be a 7 x 2 dimensional feature vector) of one parameter of the current and future 7 key point post-root points in the post-root coordinate system may first be converted to a representation in the world coordinate system and then used to update the corresponding parameters of the current and future 7 key point post-root points in the track sequence corresponding to the input x i of the motion prediction network.
By the embodiment of the application, when the action state information is updated, the track sequence is updated according to the corresponding relation between the track sequences, so that the reasonability of track sequence updating can be improved.
As an alternative embodiment, the parameters corresponding to the target point may include, but are not limited to, at least one of: position, direction, velocity vector. The position, direction and velocity vectors may all be two-dimensional vectors. Correspondingly, the set of track sequences may comprise at least one of: the first set of sub-track sequences may include at least one of the following: a position sub-track sequence, a direction sub-track sequence, and a velocity vector sub-track sequence.
For example, the output y i of the motion prediction network includes: the local coordinate representation (7x6=42 dimensions) of the front track sequence with respect to the current and future last 7 key points, the local coordinate representation (7x6=42 dimensions) of the rear track sequence with respect to the current and future last 7 key points, the local coordinate representation (7x6=42 dimensions) of the current and future last 7 key points.
Optionally, in this embodiment, updating the set of track sequences according to the first set of sub-track sequences may include at least one of:
s41, updating the values of the position parameters of the target points in the position track sequences in the first group of track sequences at the current moment and a group of future moments according to the predicted values of the position parameters of the target points in the position track sequences in the first group of track sequences at the current moment and a group of future moments;
s42, updating the values of the direction parameters of the target points in the direction track sequences in the first group of track sequences at the current moment and a group of future moments according to the predicted values of the direction parameters of the target points in the direction track sequences in the first group of track sequences at the current moment and a group of future moments;
S43, updating the values of the speed vector parameters of the target points in the speed vector track sequences in the first group of track sequences at the current moment and the future moment according to the predicted values of the speed vector parameters of the target points in the speed vector track sequences in the first group of track sequences at the current moment and the future moment.
For a sequence of position trajectories in a set of trajectory sequences, it may contain values of the position parameters of the target point at a set of historical moments, at the current moment, and at a set of future moments. Correspondingly, the position sub-track sequence in the first set of sub-track sequences may comprise predicted values of the position parameters of the target point at the current moment and at a set of future moments. When updating the position track sequence in the track sequence set, the position sub track sequence in the first sub track sequence set may be used to update only the values of the position parameter of the target point at the current time and a set of future times after the current time, i.e. the values of the position parameter of the target point in the position track sequence in the track sequence set may be updated according to the predicted values of the position parameter of the target point in the position sub track sequence in the first sub track sequence set at the current time and a set of future times, and the updating may be performed according to the corresponding position.
Optionally, when the target point includes a front target point and a rear target point, the position track sequences in the first set of track sequences may be updated in the same or similar manner as the foregoing by using the position sub-track sequences in the first set of front sub-track sequences, and the position track sequences in the second set of track sequences may be updated in the same or similar manner as the foregoing by using the position sub-track sequences in the first set of rear sub-track sequences, which are already described and will not be repeated herein.
For a sequence of directional trajectories in a set of trajectory sequences, it may contain values of the directional parameters of the target point at a set of historical moments, current moments, and a set of future moments. Correspondingly, the direction sub-track sequence in the first set of sub-track sequences may comprise predicted values of the direction parameters of the target point at the current moment and at a set of future moments. When updating the direction track sequence in the track sequence, the direction sub track sequence in the first group of sub track sequences may be used to update only the values of the direction parameters of the target point at the current time and a group of future times after the current time, i.e. the values of the direction parameters of the target point in the direction track sequence in the group of track sequences at the current time and a group of future times may be updated according to the predicted values of the direction parameters of the target point in the direction sub track sequence in the first group of sub track sequences at the current time and a group of future times, and the updating may be performed according to the corresponding directions.
Optionally, when the target point includes a front target point and a rear target point, the direction track sequence in the first set of track sequences may be updated in the same or similar manner as the foregoing by using the direction sub-track sequence in the first set of front sub-track sequences, and the direction track sequence in the second set of track sequences may be updated in the same or similar manner as the foregoing by using the direction sub-track sequence in the first set of rear sub-track sequences, which are already described and will not be repeated herein.
For a velocity vector trajectory sequence in a set of trajectory sequences, it may contain the values of the velocity vector parameters of the target point at a set of historical moments, the current moment, and a set of future moments. Correspondingly, the velocity vector sub-track sequence in the first set of sub-track sequences may comprise predicted values of the velocity vector parameters of the target point at the current moment and a set of future moments. When updating the velocity vector trajectory sequences in the set of trajectory sequences, only the values of the velocity vector parameters of the target point at the current time and a set of future times after the current time may be updated using the velocity vector sub-trajectory sequences in the set of sub-trajectory sequences, i.e. the values of the velocity vector parameters of the target point in the velocity vector trajectory sequences in the set of trajectory sequences at the current time and a set of future times may be updated according to the predicted values of the velocity vector parameters of the target point in the velocity vector sub-trajectory sequences in the first set of sub-trajectory sequences at the current time and a set of future times.
Optionally, when the target point includes a front target point and a rear target point, the velocity vector track sequence in the first set of track sequences may be updated in the same or similar manner as the foregoing by using the velocity vector sub-track sequence in the first set of front sub-track sequences, and the velocity vector track sequence in the second set of track sequences may be updated in the same or similar manner as the foregoing by using the velocity vector sub-track sequence in the first set of rear sub-track sequences, which are already described herein and will not be repeated.
Alternatively, the track portion of the action state information may include some information that is expected to be future and that has occurred in the past, where the expected future information may be changed by being later entered by the user, e.g., information in a set of track sequences corresponding to a set of future times after the current time, and regarding which portion has occurred in the past, the state that has actually occurred in the past period of time (e.g., 1 second) may be stored by some buffer for sampling of the track sequences. In this embodiment, the above-mentioned buffer may be assumed by a set of history track sequences in the action state information, where each history track sequence in the set of history track sequences includes a value of a parameter (for example, a position parameter, a direction parameter, a velocity vector parameter, etc.) corresponding to the target point at each time point between the current time point and the earliest time point in the set of history time points, the current time point, and the set of future time points, or may be a value of a value corresponding to each time point between the current time point and a preset time point before the current time point, where a time difference between the preset time point and the current time point may be a predetermined time difference, for example, 1s.
Optionally, when the target point includes a front target point and a rear target point, the set of history track sequences may include a first set of history track sequences (i.e., a set of front history track sequences) and a second set of history track sequences (i.e., a set of rear history track sequences), wherein each of the first set of history track sequences includes a value of a parameter corresponding to the front target point at each time point between a current time point and an earliest time point of the set of history time points, the current time point and the set of future time points, and each of the second set of history track sequences includes a value of a parameter corresponding to the rear target point at each time point between the current time point and the earliest time point of the set of history time points, the current time point and the set of future time points.
For example, the action state information may include a set of pre-history trace sequences and a set of post-history trace sequences, where the set of pre-history trace sequences includes: a pre-history position trace, a pre-history direction trace, a pre-history velocity vector trace, a set of post-history trace sequences comprising: a post-history position trajectory, a post-history direction trajectory, and a post-history velocity vector trajectory.
Before updating the motion state information according to the motion state reference information, the method further comprises: and removing the value corresponding to the earliest moment in one parameter corresponding to the target point in each historical track sequence in the set of historical track sequences, and putting the value of one parameter corresponding to the target point at the current moment into the set of historical track sequences.
Optionally, when the target point includes a front target point and a rear target point, removing a value corresponding to an earliest moment in one parameter corresponding to the target point in each of the set of historical track sequences, and placing the value of the one parameter corresponding to the target point at the current moment in the set of historical track sequences may include: removing the value corresponding to the earliest moment in one parameter corresponding to the target front point in each historical track sequence in the first group of historical track sequences, and putting the value of one parameter corresponding to the target front point at the current moment into the first group of historical track sequences; and removing the value of one parameter corresponding to the target rear point in each historical track sequence in the second set of historical track sequences and the value corresponding to the earliest moment, and putting the value of one parameter corresponding to the target rear point in the current moment into the second set of historical track sequences.
For example, as shown in table 1, the motion state information of the animated character may include all or part of the information shown in table 1, for example, may include a track sequence, which may include a front track sequence and a rear track sequence. The pre-track sequence may include: a front track sequence (position) (front track position sequence, i.e. position track sequence in the first set of track sequences), a front track sequence (direction) (front track direction sequence, i.e. direction track sequence in the first set of track sequences), a front track sequence (velocity vector) (front track velocity vector sequence, i.e. velocity vector track sequence in the first set of track sequences). The post-track sequence may include: rear track sequence (position) (rear track position sequence, i.e. position track sequence in the second set of track sequences); rear track sequence (direction) (rear track direction sequence, i.e., direction track sequence in the second set of track sequences); a post-track sequence (velocity vector) (post-track velocity vector sequence, i.e., velocity vector track sequence in the second set of track sequences), any of which is a 13 x 2-dimensional feature vector.
The action state information may further include: a historical track sequence, which may include a pre-historical track sequence and a post-historical track sequence. The pre-history track sequence may include: a pre-history track sequence (location) (i.e., a location track sequence in the first set of history track sequences), a pre-history track sequence (direction) (i.e., a direction track sequence in the first set of history track sequences), a pre-history track sequence (velocity vector) (i.e., a velocity vector track sequence in the first set of history track sequences). The post-history track sequence may include: a post-history track sequence (location) (i.e., a sequence of location tracks in a second set of history track sequences); a post-history track sequence (direction) (i.e., a direction track sequence in the second set of history track sequences); a historic track sequence (velocity vector) (i.e., a sequence of velocity vectors in a second set of historic track sequences), either of which is a 60 x 2 dimensional feature vector.
TABLE 1
| Variable name | Dimension(s) |
| Front track sequence (position) | 13×2 |
| Front track sequence (direction) | 13×2 |
| Front track sequence (velocity vector) | 13×2 |
| Rear track sequence (position) | 13×2 |
| Rear track sequence (direction) | 13×2 |
| Rear track sequence (velocity vector) | 13×2 |
| Track sequence (gait) | 13×5 |
| Desired speed scalar (front trajectory) | 7 |
| Desired speed scalar (rear track) | 7 |
| Joint position | 17×3 |
| X-axis measurement of joint | 17×3 |
| Amount of joint z-axis | 17×3 |
| Joint velocity vector | 17×3 |
| Historic track (position) | 60×2 |
| Historical front track (direction) | 60×2 |
| Historical front track (velocity vector) | 60×2 |
| Historical back track (position) | 60×2 |
| Historical back track (direction) | 60×2 |
| Historical back track (velocity vector) | 60×2 |
Table 1 (subsequent)
| Variable name | Dimension(s) |
| Historical track (gait) | 60×5 |
| Desired orientation (front track) | 2 |
| Desired orientation (rear track) | 2 |
| Desired velocity vector (front trajectory) | 2 |
| Desired velocity vector (rear trajectory) | 2 |
| Desired correction amount | 1 |
The post-history trajectory (position) may be a 60×2-dimensional feature vector, the post-history trajectory (direction) may be a 60×2-dimensional feature vector, the post-history trajectory (velocity vector) may be a 60×2-dimensional feature vector, and the post-history trajectory (velocity vector) may be a 60×2-dimensional feature vector, the post-history trajectory (velocity vector) may be a 60-frame post-root velocity vector, the post-history trajectory (velocity vector) may be a 60×2-dimensional feature vector, the post-root position being within 60 frames (or 60 times) of the past 1 second.
At the end of each tick, the earliest frame of data (or earliest time of day) pop in the past 60 frames of historical track sequences (a set of pre-historical track sequences and a set of post-historical track sequences) can be removed, then the current root information can be stored in a queue, and then track values in the track sequences for the past 6 sample time points can be updated.
Illustratively, as shown in fig. 9, the current frame is F 0, the history track sequence includes data of 60 frames F -60 to F -1, at the end of the current frame, the data pop of F -60 is dequeued, the root information of the current frame is put into the history track sequence, and the first six points in the update track sequence (13 key times, the front track sequence and the rear track sequence) with respect to the past time are resampled, that is, the last 6 sampling times in the update track sequence: past 1 second, 5/6 second, 4/6 second, 3/6 second, 2/6 second, 1/6 second. Next, 7 points after the track sequence may be updated, and the update formula of the track position rp may be as shown in formula (1):
rpk＝(1-fn*Tc)*rpk+fn*Tc*[(1-fv)*yrpk+fv*(y′rpk+y′rvk*dt)] (1)
Here, k=0, 1,2, …,6, the reference numerals for the last 7 points, f n and f v are all adjustable fusion parameters, and f n＝0.5,fv＝0.75;Tc can be selected as the desired correction amount for correcting the first set of front sub-track sequences and the first set of rear sub-track sequences; d t is the distance between two key moments of the track sequence, i.e. 1/6 seconds; y rpk and y rvk are representations of the corresponding portions of the output of the motion prediction network, y k, transformed into the world coordinate system, and y 'rpk and y' rpk are representations of the corresponding portions of the output of the motion prediction network, y k-1, transformed into the world coordinate system.
The updated formulas of the track sequence (direction) rd and the track sequence (speed vector) rv are shown in the formula (2) and the formula (3), respectively:
rdk＝(1-fn)*rdk+fn*yrdk (2)
rvk＝(1-fn)*rvk+fn*yrvk (3)
Here, k, f n, and y rvk have the same meaning as in equation (1), and y rdk is a representation of the corresponding part of the output y k of the motion prediction network converted into the world coordinate system. The direction rd may be updated to have a modulus length less than 1, and the modulus length itself may be adjusted after each calculation to be a unit vector with a modulus length of 1.
According to the embodiment provided by the application, the position track sequence, the direction track sequence and the motion vector track sequence in a group of track sequences are respectively updated correspondingly, so that the accuracy of information updating can be improved.
As an optional implementation manner, updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment, including:
S51, when the motion state information comprises a current joint parameter value set and the motion state reference information comprises a predicted joint parameter value set, updating the current joint parameter value set according to the predicted joint parameter value set, wherein the current joint parameter value set comprises the current value of a group of parameters corresponding to a group of joints of a controlled object, and the predicted joint parameter value set comprises the predicted value of a group of parameters corresponding to a group of joints.
In order to make the action of the controlled object more expressive, a set of joints can be further arranged to express the states of different parts of the controlled object. For different kinds of controlled objects and controlled objects with different precision, the joint types and the joint numbers of a group of joints contained in the controlled objects can be distinguished. For example, for a four-foot controlled object, there may be 3 spinal joints, 4 hip joints, 4 knee joints, 4 ankle joints, 1 tail joint, 1 head joint, for a total of 17 joints. The motion state information may include a current joint parameter value set, where the current joint parameter value set may include a current value of a set of parameters corresponding to a set of joints of the controlled object, and the set of parameters may include at least one of: position parameters, attitude vector parameters, velocity vector parameters, etc.
If the motion state information includes a current joint parameter value set and the motion state reference information includes a predicted joint parameter value set, the current joint parameter value set may be updated using the predicted joint parameter value set, where the predicted joint parameter value set includes predicted values of a set of parameters corresponding to a set of joints. The method of updating the current joint parameter value set by using the predicted joint parameter value set may be one or more, for example, direct replacement, or, for example, after performing weighted summation of the corresponding values according to the correspondence, updating by using the summed value, which is not limited in this embodiment.
According to the embodiment provided by the application, the current value of the joint parameter is updated by using the predicted value of the joint parameter, so that the capability of representing the current state of the controlled object by the state information of the controlled object can be improved, and the expressive force of the action of the controlled object is further improved.
As an alternative embodiment, the set of parameters corresponding to the joint may include at least one of: position parameters, pose vector parameters, velocity vector parameters, etc., different types of joint parameters may correspond to feature vectors of different dimensions. For example, the current 3-dimensional position (i.e., position parameter), 3-dimensional x-axis vector, 3-dimensional z-axis vector (i.e., pose vector parameter), 3-dimensional velocity vector (i.e., velocity vector parameter) of each joint in the world coordinate system may be stored in the motion state information. The length of the x-axis and z-axis measuring modes of any joint are 1 and are mutually perpendicular, the gesture can be represented by using an x-z vector, and the statistical rule is more stable compared with other gesture expression modes. When updating the current joint parameter value set according to the predicted joint parameter value set, the value update can be performed in a corresponding mode for different types of joint parameters. Here, the joint portion in the operation state information may include only information of the current time.
If the set of parameters corresponding to the joint includes a position parameter, the current value of the position parameter corresponding to the set of joints in the current joint parameter value set can be updated according to the predicted value of the position parameter corresponding to the set of joints in the predicted joint parameter value set. The updating may be to replace the current value of the corresponding position parameter with the predicted value of each position parameter, or to perform weighted summation on the predicted value of each position parameter and the current value of the corresponding position parameter, and to update the current value of the corresponding position parameter with the summed value.
If the set of parameters corresponding to the joint includes a posture vector parameter, the current value of the posture vector parameter corresponding to the set of joints in the current joint parameter value set may be updated according to the predicted value of the posture vector parameter corresponding to the set of joints in the predicted joint parameter value set. The updating may be to replace the current value of the corresponding posture vector parameter with the predicted value of each posture vector parameter, or to perform weighted summation on the predicted value of each posture vector parameter and the current value of the corresponding posture vector parameter, and update the current value of the corresponding posture vector parameter with the summed value.
If the set of parameters corresponding to the joint includes a velocity vector parameter, the current value of the velocity vector parameter corresponding to the set of joints in the current joint parameter value set may be updated according to the predicted value of the velocity vector parameter corresponding to the set of joints in the predicted joint parameter value set. The updating may be to replace the current value of the corresponding speed vector parameter with the predicted value of each speed vector parameter, or to perform weighted summation on the predicted value of each speed vector parameter and the current value of the corresponding speed vector parameter, and to update the current value of the corresponding speed vector parameter with the summed value.
For example, the output y i of the motion prediction network further includes: the local coordinate representation (feature vector of 9 x 9 = 81 dimensions) of the position and posture vector of the 9 joints of the left front leg 3 joint, the right front leg 3 joint, the front spine, the middle spine and the head under the current front root coordinate system; and the local coordinate representation (feature vector of 9×9=81 dimensions) of the position and posture vector of the 9 joints of the left rear leg 3 joint, the right rear leg 3 joint, the rear spine, the middle spine, and the tail under the current rear root coordinate system, wherein the joint of the middle spine appears twice.
When the motion state information is updated (i.e., the neural network output is fused), the output y i can be converted into a representation in the world coordinate system according to the corresponding reference coordinate system, and then fused into the current state s i according to a certain scaling factor.
When updating the joint information in the operation state information s i, the joint information may first be converted into a representation in the world coordinate system. Here, the positions and directions of the front and rear shots have been changed when the track sequence is updated, and then roots after the change may be used to perform the conversion, that is, the track sequence is updated first, and the joint information is updated using the updated track sequence. For the middle spine joint, two groups of position, posture vectors and speed vectors which are expressed under a world coordinate system can be obtained, and the two groups of joint information can be weighted and averaged to be used as the joint information of the middle spine.
For the joint position jp, the joint position jp in the motion state information s i can be updated by the formula (4):
jpk＝(1-fn)*jpk+fn*[(1-fv)*yjpk+fv*(y′jpk+y′jvk*dt)] (4)
Where k, f n、fv, and d t have the same meaning as in equation (1), y jpk and y jvk are representations of the corresponding portions of the output of the motion prediction network, y k, being transformed into the world coordinate system, and y 'jpk and y' jpk are representations of the corresponding portions of the output of the motion prediction network, y k-1, being transformed into the world coordinate system.
For the joint posture vector jd and the velocity vector jv, the joint posture vector jd and the velocity vector jv in the motion state information s i can be updated by the equation (5) and the equation (6):
jdk＝(1-fn)*jdk+fn*yjdk (5)
jvk＝(1-fn)*jvk+fn*yjvk (6)
where k, f n and y jvk have the same meaning as in equation (4), y jdk is the representation of the corresponding part of the output y k of the motion prediction network converted into the world coordinate system. The possible modular length after the direction vector jd is updated will be less than 1, and the own modular length can be adjusted after each calculation to be a unit vector with the modular length of 1. In addition, the x-axis and the z-axis of the same joint obtained at this time are not necessarily perfectly perpendicular, and the vector obtained by x-axis cross-multiplication (the result of current z-axis cross-multiplication of the x-axis) can be used for copying the z-axis, so as to ensure that the x-axis and the z-axis are perpendicular to each other.
In addition, the current state information s i obtained may also undergo some rule-based post-processing. For a quadruped model in a gaming application, the length of each bone is typically unchanged, so the joint position and joint pose in the current state information s i are not independent of each other. The result under action predicting neural network bias will be redundant and conflicting. If the skeleton model in the game animation is mainly a ball joint, the skeleton model can be put into the skeleton model in a trust joint gesture mode, the actual reached position of the joint is recalculated, and the original joint position in the current state information s i is copied.
In order to prevent the slip of the character model end (four leg ends) in contact with the ground during the intelligent game animation generation, the relative joint positions and joint angles may also be adjusted using inverse kinematics so that the end joints (i.e., ankle joints) of the legs that contact the ground are desirably maintained at positions in the same world coordinate system.
According to the embodiment provided by the application, the accuracy of updating the state information can be improved by correspondingly updating at least part of the position parameters, the gesture vector parameters and the motion vector parameters of the joint.
As an alternative embodiment, the control instruction input by the user may or may not be detected at the current moment, and thus, when generating the predicted action instruction, the predicted action instruction may be generated in a different manner based on whether the control instruction input by the user is detected.
As an optional implementation manner, generating a predicted action instruction of the controlled object at the next moment based on the action state prediction information includes:
S61, when motion control information for controlling the controlled object input by the target input device is not detected at the current moment, a predicted motion instruction of the controlled object at the next moment is generated according to the motion state prediction information.
When the control information for controlling the controlled object input by the target input device is not detected at the current moment, the user input is not detected, and the motion state estimated information can be used as the motion state information of the controlled object at the next moment. At this time, the server may directly generate a predicted action instruction of the controlled object at the next moment according to the action state prediction information.
As another alternative embodiment, generating a predicted motion instruction of the controlled object at the next moment based on the motion state prediction information includes:
S62, when motion control information input by the target input device and used for controlling the controlled object is detected at the current moment, motion state estimated information of the controlled object is updated according to the motion control information input by the target input device, and updated motion state estimated information of the controlled object is obtained;
s63, generating a predicted action instruction of the controlled object at the next moment according to the updated action state predicted information.
Optionally, the user's input, after being processed by an input processing module in the server, changes the content of the motion state information about the future trajectory. The user may input motion manipulation information for controlling the controlled object through the target input device, and the server may acquire the motion manipulation information. The target input device may be a device allowing interaction with the target application, e.g. a handle, a keyboard, a mouse, but also other input devices.
For example, for a quadruped character (i.e., a virtual character employing a quadruped model, a controlled robot, and the like), a user may control the quadruped character by inputting keyboard information, and the correspondence between control commands of the quadruped character and output animations may be as shown in table 2.
TABLE 2
| Command | Output animation |
| Press and hold W (first person name) | 1 St speed straight forward |
| Press S (first person name) | 1 St speed straight line reverse |
| The first person says that the A/D is pressed | Shift 1 speed left/right translation |
| (First person name) hold WA/WD/SA/SD | Gear 1 speed translates left front/right front/left rear/right rear |
| (First person name) hold Ctrl+W/WA/WD | Speed 2 jogging forward/left forward/right forward |
| (First person name) press Shift+W/WA/WD | 3-Speed forward/left forward/right forward sprinting |
| (First person name) hold Enter+W | 4-Gear speed forward racing |
| (First person name) hold down Q/E | 1 St speed reverse/clockwise in-situ rotation |
| (First person name) hold QW/EW | Reverse/clockwise winding at 1 st gear speed |
| (First person name) hold Ctrl+QW/EW | 2 Nd speed reverse/clockwise winding jogging |
| Press G | Switching first/third person control modes |
| (Third person calls) press the W/S/A/D | The 1 st speed is to go straight forward/backward/left/right to the user's viewing angle |
| (Third person calls) holding Ctrl+W/S/A/D | 2 Nd speed jogging forward/backward/left/right to user's view angle |
| (Third person calls) press Shift+W/S/A/D | 3 Rd speed forward/backward/left/right sprinting to user's view angle |
| (Third person calls) holding Enter+W/S/A/D | 4-Speed racing to the front/back/left/right view of the user |
| (Third person calls) holding Q/E | Turning user perspective left/right |
| Pressing Z/C (during racing) | Sudden braking and left/right turning about 135 deg |
| (When racing) pressing Space | Jumping |
Table 2 (subsequent)
| Command | Output animation |
| Press X (standing or 1 st gear) | Sit down |
| Pressing Alt (standing or 1 st gear) | Lying down |
| No key or illegal key | (Hold or transition to) quadruped standing still |
The animation effect corresponding to the key input of the user in the game (for the controlled robot, the output animation is the output action) is contained in the table 2, and the deep learning model can also intelligently generate the connection picture when different keys are switched. Compared with the key control mode of WSAD, the key control mode in table 2 can have more degrees of freedom and generate richer double-track combination, so that the four-foot drop points of the virtual character (for example, the quadruped animal) have more expressive force, and the explosive force of the waist and abdomen of the quadruped animal can be displayed more vividly.
The input of the motion operation information affects the motion state estimation information of the controlled object. If the control information for controlling the controlled object input by the target input device is detected at the current moment, the motion state estimated information of the controlled object can be updated according to the motion control information, so that the updated motion state estimated information of the controlled object is obtained. The update may be: and determining state information adjusted by the action operation information in the action state estimation information, and updating the state information adjusted by the action operation information according to the action operation information.
After the updated motion state estimated information is obtained, a predicted motion instruction of the controlled object at the next moment can be generated according to the updated motion state estimated information, for example, a predicted motion of the virtual character on the next frame of the current frame is generated according to the updated motion state estimated information, so as to obtain a predicted motion instruction, and for example, a predicted motion of the passive robot at the next moment of the current moment is generated according to the updated motion state estimated information, so as to obtain the predicted motion instruction.
According to the embodiment of the application, the flexibility of the state information adjustment of the controlled object can be improved by adjusting the action state information of the controlled object by combining the input action control information.
As an alternative embodiment, updating motion state estimation information of a controlled object according to motion control information input by a target input device includes:
S71, determining a target gait vector and a target speed scalar triggered by the motion control information according to the key identification of a group of keys when the motion control information input by the target input device comprises the key identification of the group of keys which are touched;
s72, when the motion state estimated information comprises a current gait sequence and a current speed standard subsequence in the motion state information, updating the current gait sequence and the current speed standard subsequence in the motion state estimated information respectively according to the target gait vector and the target speed scalar.
In this embodiment, the controlled object may also have multiple gaits, and the gaits of the controlled object may be represented by a multidimensional gait vector, and the dimension of the gait vector may be the same as the number of gaits possible for the controlled object. Each component in the gait vector corresponds to a gait. Alternatively, the gait vector at any instant may be a normalized (sum of components is 1) multidimensional vector (e.g., a 5-dimensional vector). Optionally, the first set of track sequences and the second set of track sequences correspond to the target anterior point and the target posterior point, respectively, i.e. there is a division of the anterior-posterior track, and in gait the whole controlled object adopts a unified gait vector.
For example, for a quadruped character, the trajectory has a division of the front root trajectory and the rear root trajectory in the position, direction and velocity vectors, but the entire quadruped character adopts a unified gait vector in gait. The gait vector is a normalized 5-dimensional vector at any instant. The 5 components of the gait vector represent respectively standing, walking (including running), jumping, sitting, lying prone. Where (1, 0) indicates pure stance and (0.8,0.2,0,0,0) indicates transition from stance to ambulatory until the vector becomes (0, 1, 0) indicates complete transition of gait to ambulatory.
The motion state estimation information of the controlled object may include a current gait sequence and a current speed scalar subsequence in the motion state information. Here, the current gait sequence includes values of a set of gait parameters of the controlled object at a set of historical moments, a current moment and a set of future moments, each gait parameter being used to represent a gait of the controlled object, which may be, for example, the aforementioned 5-dimensional gait vector, and the current gait sequence may be a 13×5-dimensional gait vector.
The current velocity scalar sub-sequence includes the values of the velocity scalar parameters of the target point at the current time and a set of future times. When the target point includes a pre-target point and a post-target point, the current speed scalar sub-sequence may include: a first velocity scalar sub-sequence corresponding to the target front point, which may include values of a velocity scalar parameter of the target front point at a current time and a set of future times, which may be a desired velocity scalar (front trajectory), i.e., a desired front trajectory velocity scalar, for example, the first velocity scalar sub-sequence may be a 7 x1 dimensional vector; the second velocity scalar sub-sequence corresponding to the target rear point, which may include the values of the velocity scalar parameters of the target rear point at the current time instant and a set of future time instants, may be a desired velocity scalar (rear track), i.e., a desired rear track velocity scalar, for example, the second velocity scalar sub-sequence may be a 7 x1 dimensional vector.
For example, in addition to the track information, the motion state information may include 7 scalar amounts of expected speed of the front track and the rear track, which respectively indicate the expected translation rate of the front root or the rear root point in the horizontal direction at the current time, the future time of 1/6 seconds, 2/6 seconds, 3/6 seconds, 4/6 seconds, 5/6 seconds, and 1 second.
If the motion manipulation information includes a key identification of a set of keys that are touched (e.g., a key identification as shown in table 2), a target gait vector and a target speed scalar triggered by the motion manipulation information may be determined based on the key identification of the set of keys. The target gait vector may comprise values of the above-mentioned set of gait parameters of the controlled subject. The motion state estimation information may further include a target speed scalar, where the target speed scalar represents a value of a speed scalar parameter of the controlled object, and may include a speed scalar corresponding to a target front point and a speed scalar corresponding to a target rear point.
The user's input may also change what is in the motion state estimation information about the desired gait. If the motion state estimation information comprises the current gait sequence and the current speed standard subsequence in the motion state information, the current gait sequence and the current speed standard subsequence in the motion state estimation information can be updated according to the target gait vector and the target speed scalar respectively. The updating may be updating gait information at a current time, a set of future times after the current time, in the current gait sequence using the target gait vector, and updating the current speed scalar subsequence using the target speed scalar, the updating may be updating the speed scalar subsequence corresponding to the target front point in the current speed scalar subsequence using the speed scalar corresponding to the target front point in the target speed scalar, and updating the speed scalar subsequence corresponding to the target rear point in the current speed scalar subsequence using the speed scalar corresponding to the target rear point in the target speed scalar.
According to the embodiment provided by the application, the expected gait and expected speed vectors of the virtual character are adjusted based on the user input, so that the flexibility of the state adjustment of the controlled object can be improved.
As an alternative embodiment, the gait vector of the controlled object may comprise a plurality of components, which may include, but is not limited to, at least one of: jump parameters, sitting parameters, lying parameters, running parameters, standing parameters. Correspondingly, when determining the target gait vector and the target speed scalar triggered by the motion manipulation information based on the key identifications of the set of keys, the corresponding target gait vector and target speed scalar may be determined for the key identifications of the different set of keys (e.g., refer to table 2).
The set of gait parameters may include a jump parameter, and when the key identification of the set of keys includes the key identification of the jump combination key or the key identification of the jump key, the gait to be adjusted may be determined to be a jump gait. At this time, the value of the jump parameter in the target gait vector may be set to a first value (e.g., 1), the value of the gait parameter other than the jump parameter in the target gait vector may be set to a second value (e.g., 0) different from the first value, and the target speed scalar may be set to a preset first speed value, which is greater than 0.
For example, if an end+w+space (an example of a key identification for a set of keys) is detected, the target gait is a (0, 1, 0) jump, the target speed scalar is 4, see table 2.
The set of gait parameters may include a sitting parameter, and the gait to be adjusted may be determined to be a sitting gait when the key identification of the set of keys includes a key identification of a sitting combination key or a key identification of a sitting key. At this time, the value of the sitting parameter in the target gait vector is set to a first value (e.g., 1), the value of the gait parameters other than the sitting parameter in the target gait vector is set to a second value (e.g., 0) different from the first value, and the target speed scalar is set to 0.
Here, when the controlled object is in the sitting state, in order to increase the degree of realism of the controlled object and avoid an unrealistic moving state (for example, a degree of realism of a manner of moving in an arbitrary direction in the sitting state is poor), the speed scalar corresponding to the sitting parameter may be configured to 0.
For example, if an X is detected for a keystroke, the target gait is (0, 1, 0) sitting and the target speed scalar is 0.
The set of gait parameters may include a down-going parameter, and when the key identification of the set of keys includes the key identification of the down-going combination key or the key identification of the down-going key, the value of the down-going parameter in the target gait vector may be set to a first value (e.g., 1), the value of the gait parameters other than the down-going parameter in the target gait vector may be set to a second value (e.g., 0) different from the first value, and the target speed scalar may be set to 0.
For example, if Alt is detected for a key press, the target gait is (0,0,0,0,1) groveling and the target speed scalar is 0.
The set of gait parameters may include a running parameter, and when the key identifier of the set of keys includes a key identifier of a running combination key or a key identifier of a running key, the value of the running parameter in the target gait vector may be set to a first value (e.g., 1), the value of the gait parameter other than the running parameter in the target gait vector may be set to a second value (e.g., 0) different from the first value, and the target speed scalar may be set to a preset second speed value, where the second speed value is greater than 0.
For example, if at least one of the key buttons WASDQE is detected, the target gait is (0, 1, 0) running, whether there is a press Ctrl, shift, or Enter is detected, if there is a press Ctrl, shift, or Enter, the target speed scalar is 2,3, 4, respectively, and if there is no press, the target speed scalar is 1.
The set of gait parameters may include a stance parameter, where the keystroke identification of the set of keystrokes includes a keystroke identification of a stance combination keystroke or a keystroke identification of a stance keystroke, the value of the stance parameter in the target gait vector may be set to a first value (e.g., 1), the value of the gait parameter in the target gait vector other than the stance parameter may be set to a second value (e.g., 0) different from the first value, and the target speed scalar may be set to 0.
For example, if a key is detected that does not contain any of the above, the target gait is (1, 0) standing and the target speed scalar is 0.
The jump parameter and the running parameter are gait parameters related to speed, and under the two gait parameters, a speed scalar corresponding to the jump parameter and the running parameter is determined according to a key mark of a group of keys or according to a default configuration.
Optionally, for a scene in which the controlled object is a virtual character, after the motion manipulation information is acquired, if the target button identifier is detected in the motion manipulation information, the display view angle of the virtual character may be switched first, a control manner of the virtual character may also be switched, and after the switching is completed, the target gait vector and the target speed scalar are determined.
For example, after obtaining the user key input, it is first detected whether there is G in the key. And if so, switching the view angle and the control mode of the first person or the third person. After the first and third person calls are switched, a 5-dimensional target track sequence and a target speed scalar are determined.
According to the embodiment provided by the application, the rationality of the state information adjustment of the controlled object can be ensured and the simulation degree of the controlled object can be improved by configuring the target motion vectors corresponding to the gaits such as the jumping parameters, the sitting parameters, the lying parameters, the running parameters, the standing parameters and the like.
As an alternative embodiment, updating the current gait sequence and the current speed scalar subsequence in the motion state estimation information according to the target gait vector and the target speed scalar respectively includes:
s81, updating values of a group of gait parameters of a controlled object in a current gait sequence at the current moment and a group of future moments according to the target gait vector;
S82, according to the target speed scalar, updating the value of the speed scalar parameter of the target point in the current speed scalar sub-sequence at the current time and a group of future time.
In this alternative example, when the current gait sequence (e.g., the gait of the last 7 key points in the trajectory sequence (gait) in table 1, which may be a 7 x 5-dimensional vector) is adjusted, the values of a set of gait parameters of the controlled object in the current gait sequence at the current time and a set of future times may be updated according to the target gait vector. For example, the value of the set of gait parameters of the controlled object at the current time may be set as the value of the set of gait parameters of the controlled object in the target gait vector, and the value of the set of gait parameters of the controlled object at a set of future times may be adjusted accordingly.
For example, the trajectory sequence (gait) (i.e., target_ gait) and the desired speed scalar in s i may be updated using the target gait vector (goal _ gait) and the target speed scalar (goal _speed). When the track sequence (gait) is updated, the track sequence (gait) is 13 5-dimensional vectors, k represents the sequence number of the 5-dimensional vectors, and the number of k is from 0 to 12, and then the update formula of the last 7 bits (k=6 to 12) in the track sequence (gait) is shown as formula (7):
when updating the current speed scalar sub-sequence, the speed scalar parameters of the target points in the current speed scalar sub-sequence can be updated according to the target speed scalar at the current time and a group of future time. For example, the value of the speed scalar parameter of the target point in the current speed scalar sub-sequence at the current time may be set to the value of the speed scalar parameter of the target point represented by the target speed scalar, and the value of the speed scalar parameter of the target point in the current speed scalar sub-sequence at a set of future times may be adjusted accordingly.
Alternatively, when the target point includes a front target point and a rear target point, the current velocity scalar sub-sequence may include a first velocity scalar sub-sequence corresponding to the front target point and a second velocity scalar sub-sequence corresponding to the rear target point, and the first velocity scalar sub-sequence and the second velocity scalar sub-sequence may be updated based on the target velocity scalar, respectively.
For the first velocity scalar sub-sequence, the values of the velocity scalar parameters of the target front points in the first velocity scalar sub-sequence (e.g., the desired velocity scalar (front trace) in table 1, which may be a 7 x 1 dimensional vector) at the current time and a set of future times may be updated according to the target velocity scalar. For example, the value of the speed scalar parameter of the target front point in the first speed target sub-sequence of the controlled object at the current time may be set to the value of the speed scalar parameter of the target front point represented by the target speed scalar, and the value of the speed scalar parameter of the target front point in the first speed target sub-sequence of the controlled object at a set of future time may be adjusted accordingly.
For a target rear point, the value of the speed scalar parameter of the target rear point in the second speed scalar sub-sequence (e.g., the desired speed scalar (rear track) in table 1, which may be a7 x 1 dimensional vector) may be updated at the current time and at a set of future times based on the target speed scalar. For example, the value of the speed scalar parameter of the target rear point in the second speed scalar sub-sequence of the controlled object at the current time may be set to the value of the speed scalar parameter of the target rear point represented by the target speed scalar, and the value of the speed scalar parameter of the target rear point in the second speed scalar sub-sequence of the controlled object at a set of future time may be adjusted accordingly.
Illustratively, when updating the desired speed scalar (which may be, for example, a desired speed scalar (front trace) and a desired speed scalar (rear trace)) using the target gait vector (goal _ gait) and the target speed scalar (goal _speed), the desired speed scalar may be updated using equation (8), where the desired speed scalar has only 7 bits, and k is numbered from 0 to 6:
according to the embodiment provided by the application, the value of a group of gait parameters of the controlled object at the current moment and a group of future moments is respectively adjusted through the target gait vector and the target speed scalar, and the value of the speed scalar parameter of the target point of the controlled object at the current moment and a group of future moments is adjusted, so that the rationality and the flexibility of gait and speed vector adjustment can be improved.
As an alternative embodiment, updating motion state estimation information of a controlled object according to motion control information input by a target input device includes:
s91, determining a target rotation angle of a controlled object according to a key identification of a group of keys;
S92, determining a target fusion coefficient according to the target gait vector;
S93, when the motion state estimated information comprises the current expected direction and the current expected speed vector in the motion state information, updating the current expected direction in the motion state estimated information according to the target rotation angle and the target fusion coefficient, and updating the current expected speed vector in the motion state estimated information according to the target speed scalar and the target fusion coefficient.
In this embodiment, the motion state estimation information may include a current desired direction (for example, a desired direction (front trajectory) and a desired direction (rear trajectory) in table 1, which may be two 2-dimensional vectors) and a current desired speed vector (for example, a desired speed vector (front trajectory) and a desired speed vector (rear trajectory) in table 1, which may be two 2-dimensional vectors) in the motion state information. The current desired direction and the current desired speed vector may both be two-dimensional vectors, and the current desired speed vector may be a unit vector with a modulo length of 1. When the target point includes a front target point and a rear target point, the current desired orientation may include a current desired orientation of the front target point and a current desired orientation of the rear target point, and the current desired velocity vector may include a current desired velocity vector of the front target point and a current desired velocity vector of the rear target point.
According to the key identification of a group of keys, the target rotation angle of the controlled object can be determined, wherein the rotation angle can be an angle rotating around the z-axis, and the target rotation angle can be used for determining the target direction of a track sequence. This target direction is a 2-dimensional unit vector on the xy-level.
For example, for a virtual character in a gaming application, based on WASDQEZC keys, an angle of rotation about the Z-axis can be calculated (e.g., 135 degrees to the left for Z and 90 degrees to the right for D for a galloping). If the view is controlled at the first person, this rotation may be added in the current front root direction; if the third person calls for controlling the viewing angle, this rotation will be applied in the camera direction, whereby a 2-dimensional object, goal dir, can be obtained.
From the target gait vector, a target fusion coefficient may be determined. Different gait vectors can be configured with corresponding fusion coefficients, and the fusion coefficients corresponding to the different gait vectors can be the same or different. For example, if the target gait is (0, 1, 0) or (0, 1, 0), the fusion coefficient factor=0.25 is determined, and if there are three other gaits, factor=0.05.
If the motion state estimation information comprises the current expected direction and the current expected speed vector in the motion state information, updating the current expected direction in the motion state estimation information according to the target rotation angle and the target fusion coefficient, and updating the current expected speed vector in the motion state estimation information according to the target speed scalar and the target fusion coefficient.
For example, for a 2-dimensional desired direction (target_dir) in the action state information, the desired direction may be updated using the target direction goal _dir and the fusion coefficient factor using equation (9):
target_dir＝(1-factor)*target_dir+factor*goal_dir (9)
For a 2-dimensional desired velocity vector (target_vel) in the motion state information, the desired velocity vector can be updated using the target velocity vector goal _speed and the fusion coefficient factor using equation (10):
target_vel＝(1-factor)*target_vel+factor*goal_speed (10)
it should be noted that the target may be changed instantaneously. Typically, at the instant of the user's change, the target is allowed to change abruptly at the current tick. While it is expected to be part of the state of the virtual character, which varies with the object, it is expected that only gradual changes are allowed, and abrupt changes are not allowed, in order to ensure smooth animation effects.
According to the embodiment provided by the application, the current expected direction is updated according to the target rotation angle and the target fusion coefficient, and the current expected speed vector is updated according to the target speed scalar and the target fusion coefficient, so that the method and the device can be suitable for angle rotation of different controlled objects under different gaits, and the simulation degree of motion generation of the controlled objects is improved.
As an alternative implementation, the current gait sequence and the current speed scalar subsequence in the motion state estimation information may be updated according to the target gait vector and the target speed scalar, respectively, in the same or similar manner as in the previous examples. Updating the current speed scalar subsequence to obtain an updated current speed scalar subsequence. Alternatively, if the current speed scalar sub-sequence includes the first speed scalar sub-sequence and the second speed scalar sub-sequence, the sub-sequence obtained by updating the first speed scalar sub-sequence is a third speed scalar sub-sequence (for example, the updated desired speed scalar (front track) may be a 7×1-dimensional vector), and the sub-sequence obtained by updating the second speed scalar sub-sequence is a fourth speed scalar sub-sequence (for example, the updated desired speed scalar (rear track) may be a 7×1-dimensional vector).
When the motion state estimated information of the controlled object is updated according to the motion control information, after updating the expected direction and the expected speed vector, if the current expected speed vector comprises a first expected speed vector corresponding to the target point and the first expected speed vector is updated to be a second expected speed vector, the value of the speed scalar parameter of the target point in the updated current speed scalar subsequence at the current time and a set of future time can be updated according to the target fusion coefficient and the second expected speed vector.
Alternatively, when the target point includes a front target point and a rear target point, the first desired speed vector may include a first front desired speed vector (for example, a desired speed vector (front track) in table 2, which may be a 2-dimensional vector) corresponding to the front target point and a first rear desired speed vector (for example, a desired speed vector (rear track) in table 2, which may be a 2-dimensional vector) corresponding to the rear target point, and the second desired speed vector includes a second front desired speed vector corresponding to the front target point and a second rear desired speed vector corresponding to the rear target point, and the scalar values of the speed scalar parameters of the front target point in the third speed scalar sub-sequence at the current time and a set of future times may be updated according to the target fusion coefficient and the second rear desired speed vector, and the scalar values of the speed scalar parameters of the rear target point in the fourth speed scalar sub-sequence at the current time and a set of future times may be updated according to the target fusion coefficient and the second front desired speed vector.
For example, the pre-track, post-track desired speed scalar may be updated again using equation (11):
speedk＝(1-factor)*speedk+factor*||target_vel|| (11)
By the embodiment provided by the application, after the expected direction and the expected speed vector are updated, the front track and the back track expected speed scalar are updated again, so that the updating reasonability of the speed scalar can be improved.
As an alternative embodiment, updating motion state estimation information of a controlled object according to motion control information input by a target input device includes:
S101, when the motion state estimated information comprises the current expected correction amount in the motion state information, updating the current expected correction amount in the motion state estimated information according to the target fusion coefficient.
If the motion state estimation information includes a current desired correction amount in the motion state information (which may be used when updating the position sub-track sequence in the first set of sub-track sequences), the current desired correction amount may also be updated according to the motion manipulation information input by the target input device. The target fusion coefficient may be used to update the current desired correction amount.
For example, the current desired correction amount (target_cor) may be updated according to equation (12):
target_cor＝(1-factor)*target_cor+factor*b (12)
Wherein, the value of b is shown in the formula (13):
The desired direction, the desired velocity vector, and the desired correction amount in table 2 may be intermediate amounts for processing user inputs. The user's input is directed to a target (goal) immediately after which the desired (target) in the state will follow the target, after which the trajectory of the controlled object is allowed to follow the desired. Since the user's input may be non-smooth, discrete, or even a high frequency oscillating input. If the character's trajectory is directly allowed to follow the target, oscillations of the model will be induced. The track following expectation is used, the target is expected to be followed, which is equivalent to the amount damping (damping) introduced by a second-order system in the control theory, so that the smooth vibration reduction effect can be achieved, and the user can feel smoother in operation.
According to the embodiment provided by the application, the expected correction coefficient is updated based on the fusion coefficient, so that the consistency of the expected correction coefficient with the current expected direction and the expected speed can be ensured, and the accuracy of the action generation of the controlled object is further improved.
As an alternative embodiment, the track sequence information comprises a set of track sequences, each track sequence of the set of track sequences comprising the value of a parameter corresponding to the target point at a set of historical moments, the current moment and a set of future moments. The values of a parameter corresponding to the target point in each track sequence in the track sequence set at the current time and the future time may be updated according to the motion state reference information in the same or similar manner as in the previous embodiment, so as to obtain a second sub-track sequence set. The motion state estimation information may include the second set of sub-track sequences described above.
Alternatively, when the target point includes a pre-target point and a post-target point, the set of track sequences may include a first set of track sequences and a second set of track sequences. The values of one parameter corresponding to the target front point in each track sequence in the first set of track sequences at the current time and a set of future time may be updated according to the motion state reference information, so as to obtain a second set of front sub-track sequences (for example, the front track sequence (position), the front track sequence (direction), and the data corresponding to the rear 7 key points in the front track sequence (speed vector) in table 1 may be 7×6-dimensional vectors). The values of one parameter corresponding to the target rear point in each track sequence in the second set of track sequences at the current time and a set of future time may be updated according to the motion state reference information, so as to obtain a second set of rear sub-track sequences (for example, the data corresponding to the rear 7 key points in the rear track sequence (position), the rear track sequence (direction) and the rear track sequence (speed vector) in table 1 may be 7×6-dimensional vectors). The second set of sub-track sequences may include the second set of front sub-track sequences and the second set of rear sub-track sequences described above.
When the motion state estimation information is updated according to the motion control information, the values of the position parameters of the target points in the position sub-track sequences in the second group of sub-track sequences at a set of future time points can be updated according to the target expected direction and the target expected speed vector, the values of the direction parameters of the target points in the direction sub-track sequences in the second group of sub-track sequences at a set of future time points can be updated, and the values of the speed vector parameters of the target points in the speed vector sub-track sequences in the second group of sub-track sequences at a set of future time points can be updated.
Alternatively, when the target point includes a target front point and a target rear point, for the target front point, the values of the position parameters of the target front point in the position sub-track sequence in the second set of front sub-track sequences (for example, the front track sequence (position) in table 1) at a set of future times may be updated according to the target desired orientation (for example, the desired orientation (front track) in table 1, which may be a 2-dimensional vector) and the target desired velocity vector (for example, the velocity vector (front track) in table 1, which may be a 2-dimensional vector), respectively, and the values of the position parameters of the target front point in the position sub-track sequence in the second set of front sub-track sequences at a set of future times may be updated, and the values of the velocity vector parameters of the target front point in the velocity sub-track sequence in the second set of front sub-track sequences (for example, the front track sequence (velocity vector) in table 1) at a set of future times may be updated.
For the target rear point, the values of the position parameters of the target rear point in the second set of rear sub-track sequences (for example, the rear track sequence (position) in table 1) at a set of future times may be updated according to the target desired direction (for example, the desired direction (rear track) in table 1, which may be a 2-dimensional vector) and the target desired velocity vector (for example, the velocity vector (rear track) in table 1, which may be a 2-dimensional vector), respectively, and the values of the velocity vector parameters of the target rear point in the second set of rear sub-track sequences (for example, the rear track sequence (velocity vector) in table 1) at a set of future times may be updated.
For example, the position (pos), direction (dir), velocity vector (vel) of the last 6 bits in the 13-dimensional front track sequence and the rear track sequence may be processed separately, and as shown in fig. 10, the track sequence itself contains 13 2-dimensional vectors, numbered from 0 to 12, and when the user inputs are fused, only the last 6 bits related to the future time are affected. For the front track sequence (position) and the rear track sequence (position), the position at a certain time point is determined by the position at the previous time point, the position difference between the current time point and the previous time point, and the expected speed vector, for the front track sequence (direction) and the rear track sequence (direction), the direction at a certain time point can be determined according to the direction and the expected direction at the current time point, and for the front track sequence (speed vector) and the rear track sequence (speed vector), the speed vector at a certain time point can be determined according to the speed vector and the expected speed vector at the current time point.
By using the embodiment provided by the application, the position, the direction and the speed vector of the track sequence are adjusted by using the expected direction and the expected speed, so that the sensitivity of generating the action of the controlled object through the control operation can be improved.
The action instruction generation method in the embodiment of the present application is explained below in conjunction with alternative examples. In this alternative example, the controlled object is a quadruped model (virtual character), the target point includes a front root point and a rear root point, and the model used for performing motion prediction generates a model for the motion shown in fig. 6, which may include: the motion prediction network and the gate network, the front point of the target is the front root point, the rear point of the target is the rear root point, and a set of historical time, current time and a set of future time are 13 time points of 1 second, 5/6 seconds, 4/6 seconds, 3/6 seconds, 2/6 seconds, 1/6 seconds, current time, 1/6 seconds, 2/6 seconds, 3/6 seconds, 4/6 seconds, 5/6 seconds and 1 second.
The alternative example provides a scheme for realizing intelligent motion generation of the quadruped based on a double-track mode, which is used for generating the motion of the quadruped under the control of a user in real time, and can generate high-quality quadruped animation which can swing in a four-leg diversity mode according to the real-time control of the user in a game, so that low-quality animation in which animal bodies fly in the air and one or more legs are dragged away is avoided, the generated quadruped animation is more vivid, the manpower required by game development can be saved, and the game storage and running memory are saved.
Referring to fig. 11, 12 and 13, the flow of the action instruction generation method in this alternative example may include the steps of:
In step S1302, the neural network inputs the extraction.
When the program starts, the quadruped is initialized to stand still above the origin when the tick ordinal i=0, all track sequences, history caches and future expectations are duplicated and initialized according to the state at the moment in combination with table 1, the joint information is converted according to the current standing posture, and the expected speed and the expected correction amount are initialized by 0.
Before each tick begins, the game program extracts the corresponding information from the current state s i to render the game screen and play it in the game interface.
In each tick, information may first be extracted from the current state s i and put into the format of the input x i of the action prediction network and the gate network. X i of the motion prediction network is 451-dimensional vector, which is respectively:
the position, direction and speed vector of the front track sequence are converted into a local coordinate representation (13×6=78 dimensions) under the current front root coordinate system;
the position, direction and speed vector of the back track sequence are converted into a local coordinate representation (13×6=78 dimensions) under the current back root coordinate system;
gait sequence scalar immobilized with desired speed (13×5+7+7=79 dimensions);
The position, posture vector and speed vector of the left front leg 3 joint, the right front leg 3 joint, the front spine, the middle spine and the head 9 joints are converted into a local coordinate representation (9×12=108 dimensions) under the current front root coordinate system;
the position, posture vector, and velocity vector of the left hindleg 3 joint, right hindleg 3 joint, posterior spine, middle spine, and tail 9 joints are converted into a local coordinate representation (9×12=108 dimensions) in the current posterior root coordinate system.
In the 451-dimensional vector, there are 19-dimensional components that are input to the gate network at the same time, and these 19-dimensional components are: the speed of the left front ankle joint and the right front ankle joint under a front root local coordinate system is respectively 3 dimensions; the speed of the left rear ankle joint and the right rear ankle joint under a rear root local coordinate system is respectively 3 dimensions; a gait vector 5-dimensional at the current time and a current expected front track speed scalar; the post-track velocity scalar is 1-dimensional each. After the 451-dimensional vector and 19-dimensional component are obtained, neural network input extraction may be completed.
In step S1304, the extracted inputs are input to the motion prediction network and the gate network, respectively, and the motion prediction network performs forward computation to obtain an output y i.
In step S1306, the neural network outputs fusion.
The output y i may be converted to a representation in the world coordinate system according to the corresponding reference coordinate system, and then fused into the current state s i according to a scaling factor. The fusion here includes the fusion of the following information: updating the position, direction and speed vectors of 7 points after the track sequence; and updating joint positions, joint posture vectors and speed vectors of 7 points after the track sequence.
Step S1308, the key inputs are fused.
If the user input fusion is not available, the action generating model can also be operated by self iteration, and the original action is kept to be continuously played. To generate actions under user control, the current user's keyboard information u i may be read by, for example, an input fusion module, and then the position, direction, velocity vector, gait, and other desired values for the 6 future points in the anterior and posterior trajectory sequence in s i are unbiased in a manner similar to the fusion of y i into s i.
When key input fusion is carried out, a track sequence gait target and a speed scalar target can be determined, a desired track sequence gait (the rear 7 bits of a track sequence (gait)) and a desired speed scalar sequence are determined, and the desired track sequence gait and the desired speed scalar in s i are updated by using the target gait vector and the target speed scalar; determining a target direction of the track sequence; determining a desired direction, a desired velocity vector, and a desired correction amount of the track sequence; the speed, direction, position of the last 6 bits in the track sequence are determined.
Further, there is a requirement of "lock orientation" for the desired orientation, the desired velocity vector, and the desired correction amount in table 1. After the above processing is completed, the front track and the rear track sequence (direction) can be restored to the state before the desired direction, the desired velocity vector and the desired correction amount of the track sequence are determined. Thus, one tick period is completed.
Through the selectable example, the animation is generated for the quadruped based on double-track intelligence in real time, and the motion control of the quadruped is performed by fusing key information of a user, so that the generated animation has better expressive force, and the waist and abdomen strength feeling of the quadruped model can be displayed more vividly.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.
According to another aspect of the embodiment of the present application, there is also provided an action instruction generating apparatus for implementing the action instruction generating method described above. Fig. 14 is a block diagram of an alternative action instruction generating apparatus according to an embodiment of the present application, as shown in fig. 14, the apparatus may include:
An obtaining unit 1402, configured to obtain motion state information of a limb of a controlled object at a current time for a corresponding target point, where the motion state information includes track sequence information of the target point at a set of historical time, the current time, and a set of future time;
a prediction unit 1404, configured to perform motion state prediction based on the motion state information, so as to obtain motion state reference information of the target point at the next moment;
An updating unit 1406, configured to update the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment;
the generating unit 1408 is configured to generate a predicted motion instruction of the controlled object at the next moment based on the motion state estimation information.
It should be noted that, the acquiring unit 1402 in this embodiment may be used to perform the above-described step S302, the predicting unit 1404 in this embodiment may be used to perform the above-described step S304, the updating unit 1406 in this embodiment may be used to perform the above-described step S306, and the generating unit 1408 in this embodiment may be used to perform the above-described step S308.
According to the embodiment of the application, the problem that the storage space and the running space are occupied greatly due to excessive required action materials in the action instruction generation method in the related art is solved based on the mode of action generation of the corresponding relation between the limb pair and the target point, the occupation of the storage space and the running space is reduced, and the simulation degree of action generation is improved.
As an alternative embodiment, the acquisition unit 1402 includes:
The first acquisition module is used for acquiring a first group of track sequences of the front point of the target corresponding to the first limb pair of the controlled object at the current moment, wherein each track sequence in the first group of track sequences comprises the values of one parameter corresponding to the front point of the target at a group of historical moment, the current moment and a group of future moments;
The second acquisition module is used for acquiring a second group of track sequences of the target rear point corresponding to the second limb pair of the controlled object at the current moment, wherein each track sequence in the second group of track sequences comprises values of one parameter corresponding to the target rear point at a group of historical moment, the current moment and a group of future moment;
The limb pairs of the controlled object comprise a first limb pair and a second limb pair, the target point comprises a target front point and a target rear point, and the track sequence information comprises a first group of track sequences and a second group of track sequences.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, prediction unit 1404 includes:
The input module is used for inputting the motion state information into the target prediction neural network model to obtain the output motion state reference information, wherein the motion state reference information comprises a first group of sub-track sequences, and each sub-track sequence in the first group of sub-track sequences comprises a predicted value of a parameter corresponding to the target point at the current moment and a group of future moments.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the track sequence information includes a set of track sequences, each track sequence in the set of track sequences including a value of a parameter corresponding to the target point at a set of historical time instants, a current time instant, and a set of future time instants; the updating unit 1406 includes:
the first updating module is used for updating a group of track sequences according to a first group of sub-track sequences, wherein the predicted value of a parameter corresponding to a target point in each sub-track sequence in the first group of sub-track sequences at the current moment and a group of future moments is used for updating the value of a parameter corresponding to the target point in each track sequence in the group of track sequences at the current moment and a group of future moments.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the first updating module comprises at least one of the following:
the first updating sub-module is used for updating the values of the position parameters of the target points in the position track sequences in the group at the current moment and the future moment according to the predicted values of the position parameters of the target points in the position track sequences in the first group at the current moment and the future moment;
the second updating sub-module is used for updating the values of the direction parameters of the target points in the direction track sequences in the group at the current moment and the future moment according to the predicted values of the direction parameters of the target points in the direction track sequences in the first group at the current moment and the future moment;
and the third updating sub-module is used for updating the values of the speed vector parameters of the target points in the speed vector track sequences in the group according to the predicted values of the speed vector parameters of the target points in the speed vector track sequences in the first group in the current time and the group in the future time.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the updating unit 1406 includes:
The second updating module is used for updating the current joint parameter value set according to the predicted joint parameter value set when the motion state information comprises the current joint parameter value set and the motion state reference information comprises the predicted joint parameter value set, wherein the current joint parameter value set comprises the current value of a group of parameters corresponding to a group of joints of the controlled object, and the predicted joint parameter value set comprises the predicted value of a group of parameters corresponding to a group of joints.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the second updating module comprises at least one of:
a fourth updating sub-module, configured to update, according to the predicted values of the position parameters corresponding to the one set of joints in the predicted joint parameter value set, the current values of the position parameters corresponding to the one set of joints in the current joint parameter value set;
a fifth updating sub-module, configured to update, according to the predicted values of the posture vector parameters corresponding to the set of joints in the predicted joint parameter value set, the current values of the posture vector parameters corresponding to the set of joints in the current joint parameter value set;
And the sixth updating sub-module is used for updating the current value of the speed vector parameter corresponding to one group of joints in the current joint parameter value set according to the predicted value of the speed vector parameter corresponding to one group of joints in the predicted joint parameter value set.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the generating unit 1408 includes:
The first generation module is used for generating a predicted action instruction of the controlled object at the next moment according to the action state prediction information when the action control information which is input by the target input device and used for controlling the controlled object is not detected at the current moment;
The third updating module is used for updating the motion state estimated information of the controlled object according to the motion control information input by the target input device when the motion control information input by the target input device for controlling the controlled object is detected at the current moment, so as to obtain updated motion state estimated information of the controlled object; and the second generation module is used for generating a predicted action instruction of the controlled object at the next moment according to the updated action state estimated information.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the third updating module includes:
The first determining submodule is used for determining a target gait vector triggered by the motion control information and a target speed scalar according to the key identification of a group of keys when the motion control information input by the target input device comprises the key identification of the group of keys which are touched, wherein the target gait vector comprises the value of a group of gait parameters of a controlled object, each gait parameter in the group of gait parameters represents one gait of the controlled object, and the target speed scalar represents the value of the speed scalar parameter of the controlled object;
and a seventh updating sub-module, configured to update, when the motion state estimation information includes a current gait sequence and a current speed scalar sub-sequence in the motion state information, the current gait sequence and the current speed scalar sub-sequence in the motion state estimation information according to the target gait vector and the target speed scalar respectively, where the current gait sequence includes values of a set of gait parameters of the controlled object at a set of historical time, a current time and a set of future time, and the current speed scalar sub-sequence includes values of speed scalar parameters of the target point at the current time and a set of future time.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the first determining sub-module comprises:
A first setting subunit, configured to set, when the key identifier of the set of keys includes the key identifier of the jump combination key or the key identifier of the jump key, a value of a jump parameter in the target gait vector to a first value, set a value of a gait parameter other than the jump parameter in the target gait vector to a second value different from the first value, and set the target speed scalar to a preset first speed value, where the set of gait parameters includes the jump parameter, and the first speed value is greater than 0;
A second setting subunit configured to set, when the key identifier of the set of keys includes the key identifier of the sitting combination key or the key identifier of the sitting key, a value of a sitting parameter in the target gait vector to a first value, set a value of a gait parameter other than the sitting parameter in the target gait vector to a second value different from the first value, and set the target speed scalar to 0, where the set of gait parameters includes the sitting parameter;
A third setting subunit, configured to set, when the key identifier of the set of keys includes a key identifier of the groveling combination key or a key identifier of the groveling key, a value of a groveling parameter in the target gait vector to a first value, set a value of a gait parameter other than the groveling parameter in the target gait vector to a second value different from the first value, and set the target speed scalar to 0, where the set of gait parameters includes the groveling parameter;
A fourth setting subunit, configured to set, when the key identifier of the set of keys includes a key identifier of the running combination key or a key identifier of the running key, a value of a running parameter in the target gait vector to a first value, set a value of a gait parameter in the target gait vector other than the running parameter to a second value different from the first value, and set the target speed scalar to a preset second speed value, where the set of gait parameters includes the running parameter, and the second speed value is greater than 0;
A fifth setting subunit configured to set, when the key identifier of the set of keys includes the key identifier of the standing combination key or the key identifier of the standing key, a value of a standing parameter in the target gait vector to a first value, a value of a gait parameter other than the standing parameter in the target gait vector to a second value different from the first value, and a target speed scalar to 0, where the set of gait parameters includes the standing parameter.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the seventh update sub-module includes:
a first updating subunit, configured to update, according to the target gait vector, values of a set of gait parameters of the controlled object in the current gait sequence at a current time and a set of future times;
And the second updating subunit is used for updating the values of the speed scalar parameters of the target points in the current speed scalar subsequence at the current moment and a group of future moments according to the target speed scalar.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the third updating module includes:
the second determining submodule is used for determining the target rotation angle of the controlled object according to the key identification of a group of keys;
the third determining submodule is used for determining a target fusion coefficient according to the target gait vector;
And the eighth updating sub-module is used for updating the current expected direction in the motion state estimated information according to the target rotation angle and the target fusion coefficient when the motion state estimated information comprises the current expected direction and the current expected speed vector in the motion state information, and updating the current expected speed vector in the motion state estimated information according to the target speed scalar and the target fusion coefficient.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the seventh update sub-module includes: a third updating subunit, configured to update, according to the target gait vector, values of a set of gait parameters of the controlled object in the current gait sequence at a current time and a set of future times; a fourth updating subunit, configured to update, according to the target speed scalar, the value of the speed scalar parameter of the target point in the current speed scalar subsequence at the current time and a set of future times, to obtain an updated current speed scalar subsequence; a third update module, comprising: and a ninth updating sub-module, configured to update, when the current desired velocity vector includes a first desired velocity vector corresponding to the target point and the first desired velocity vector is updated to a second desired velocity vector, a value of a velocity scalar parameter of the target point in the updated current velocity scalar sub-sequence at the current time and a set of future times according to the target fusion coefficient and the second desired velocity vector.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the third updating module includes:
And a tenth updating sub-module, configured to update, when the motion state estimation information includes a current expected correction amount in the motion state information, the current expected correction amount in the motion state estimation information according to the target fusion coefficient.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
As an alternative embodiment, the track sequence information includes a set of track sequences, each track sequence in the set of track sequences including a value of a parameter corresponding to the target point at a set of historical time instants, a current time instant, and a set of future time instants; a third update module, comprising:
An eleventh updating sub-module, configured to, when the motion state estimation information includes a second group of sub-track sequences and the current expected direction in the motion state estimation information is updated to a target expected direction, and the current expected speed vector in the motion state estimation information is updated to a target expected speed vector, respectively update values of position parameters of target points in position sub-track sequences in the second group of sub-track sequences at a set of future times according to the target expected direction and the target expected speed vector, update values of direction parameters of target points in direction sub-track sequences in the second group of sub-track sequences at a set of future times, and update values of speed vector parameters of target points in speed vector sub-track sequences in the second group of sub-track sequences at a set of future times;
the second group of sub-track sequences are sequences obtained by updating the value of a parameter corresponding to a target point in each track sequence in the group of track sequences at the current moment and a group of future moments according to the action state reference information.
An optional example of this embodiment may refer to an example shown in the above-described action instruction generation method, and will not be described herein.
According to still another aspect of the embodiment of the present application, there is also provided an electronic device for implementing the above-mentioned action instruction generation method, where the electronic device may be a terminal device or a server as shown in fig. 1 and 2. The present embodiment is described taking the electronic device as a terminal device as an example. As shown in fig. 15, the electronic device comprises a memory 1502 and a processor 1504, the memory 1502 having stored therein a computer program, the processor 1504 being arranged to perform the steps of any of the method embodiments described above by means of the computer program.
Alternatively, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of the computer network.
Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:
S1, acquiring action state information of limbs of a controlled object at a current moment corresponding to a target point, wherein the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment;
S2, performing motion state prediction based on the motion state information to obtain motion state reference information of the target point at the next moment;
s3, updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment;
s4, based on the motion state estimated information, generating a predicted motion instruction of the controlled object at the next moment.
Alternatively, it will be understood by those skilled in the art that the structure shown in fig. 15 is only schematic, and the electronic device may also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, a palm computer, and terminal devices such as MID, PAD, etc. Fig. 15 is not limited to the structure of the electronic device described above. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 15, or have a different configuration than shown in FIG. 15.
The memory 1502 may be used to store software programs and modules, such as program instructions/modules corresponding to the action instruction generating method and apparatus in the embodiments of the present application, and the processor 1504 executes the software programs and modules stored in the memory 1502 to perform various functional applications and data processing, that is, implement the action instruction generating method described above. The memory 1502 may include high-speed random access memory, but may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1502 may further include memory located remotely from the processor 1504, which may be connected to the terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. Wherein the memory 1502 may be used for, but is not limited to, serializing files and compiling files. As an example, as shown in fig. 15, the memory 1502 may include, but is not limited to, the acquisition unit 1402, the prediction unit 1404, the update unit 1406, and the generation unit 1408 in the operation instruction generation apparatus. In addition, other module units in the action instruction generating apparatus may be included, but are not limited to, and are not described in detail in this example.
Optionally, the transmission device 1506 is configured to receive or transmit data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1506 includes a network adapter (Network Interface Controller, NIC) that may be connected to other network devices and routers via a network cable to communicate with the internet or a local area network. In one example, the transmission device 1506 is a Radio Frequency (RF) module that is configured to communicate wirelessly with the internet.
In addition, the electronic device further includes: a display 1508 for displaying a game interface of the target game; and a connection bus 1510 for connecting the respective module parts in the above-described electronic device.
In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the plurality of nodes through a network communication. Among them, the nodes may form a Peer-To-Peer (P2P) network, and any type of computing device, such as a server, a terminal, etc., may become a node in the blockchain system by joining the Peer-To-Peer network.
According to one aspect of the present application, there is provided a computer program product comprising a computer program/instruction containing program code for executing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1609, and/or installed from the removable media 1611. When executed by the central processor 1601, performs various functions provided by embodiments of the present application. The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
Fig. 16 schematically shows a block diagram of a computer system of an electronic device for implementing an embodiment of the application. As shown in fig. 16, the computer system 1600 includes a central processing unit 1601 (Central Processing Unit, CPU) that can perform various appropriate actions and processes according to a program stored in a Read-Only Memory 1602 (ROM) or a program loaded from a storage portion 1608 into a random access Memory 1603 (Random Access Memory, RAM). In the random access memory 1603, various programs and data necessary for the system operation are also stored. The cpu 1601, the rom 1602, and the ram 1603 are connected to each other by a bus 1604. An Input/Output interface 1605 (i.e., an I/O interface) is also connected to bus 1604.
The following components are connected to the input/output interface 1605: an input portion 1606 including a keyboard, a mouse, and the like; an output portion 1607 including a Cathode Ray Tube (CRT), a Liquid crystal display (Liquid CRYSTAL DISPLAY, LCD), and a speaker, etc.; a storage section 1608 including a hard disk or the like; and a communication section 1609 including a network interface card such as a local area network card, a modem, or the like. The communication section 1609 performs communication processing via a network such as the internet. The driver 1610 is also connected to the input/output interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 1610 so that a computer program read out therefrom is installed into the storage section 1608 as needed.
In particular, the processes described in the various method flowcharts may be implemented as computer software programs according to embodiments of the application. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1609, and/or installed from the removable media 1611. When executed by the central processor 1601, performs various functions defined in the system of the present application.
It should be noted that, the computer system 1600 of the electronic device shown in fig. 16 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present application.
According to one aspect of the present application, there is provided a computer-readable storage medium, from which a processor of a computer device reads the computer instructions, the processor executing the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the above embodiments.
Alternatively, in the present embodiment, the above-described computer-readable storage medium may be configured to store a computer program for performing the steps of:
S1, acquiring action state information of limbs of a controlled object at a current moment corresponding to a target point, wherein the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment;
S2, performing motion state prediction based on the motion state information to obtain motion state reference information of the target point at the next moment;
s3, updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment;
s4, based on the motion state estimated information, generating a predicted motion instruction of the controlled object at the next moment.
Alternatively, in this embodiment, it will be understood by those skilled in the art that all or part of the steps in the methods of the above embodiments may be performed by a program for instructing a terminal device to execute the steps, where the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.
The integrated units in the above embodiments may be stored in the above-described computer-readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing one or more computer devices (which may be personal computers, servers or network devices, etc.) to perform all or part of the steps of the method described in the embodiments of the present application.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In several embodiments provided by the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.
Claims (17)
1. A method of generating an action instruction, comprising:
Acquiring action state information of limbs of a controlled object on a corresponding target point at a current moment, wherein the action state information comprises track sequence information of the target point at a group of historical moment, current moment and a group of future moment;
inputting the motion state information into a target prediction neural network model to obtain output motion state reference information, wherein the motion state reference information comprises a first group of sub-track sequences, and each sub-track sequence in the first group of sub-track sequences comprises a predicted value of a parameter corresponding to the target point at the current moment and a group of future moments;
Updating the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at the next moment, wherein the method comprises the following steps: updating the current joint parameter value set according to the predicted joint parameter value set when the motion state information comprises the current joint parameter value set and the motion state reference information comprises the predicted joint parameter value set, wherein the current joint parameter value set comprises the current value of a group of parameters corresponding to a group of joints of the controlled object, and the predicted joint parameter value set comprises the predicted value of the group of parameters corresponding to the group of joints;
and generating a predicted action instruction of the controlled object at the next moment based on the action state estimated information.
2. The method according to claim 1, wherein the obtaining the motion state information of the limb pair of the controlled object for the corresponding target point at the current moment includes:
Acquiring a first group of track sequences of a target front point corresponding to a first limb pair of the controlled object at the current moment, wherein each track sequence in the first group of track sequences comprises values of a parameter corresponding to the target front point at the historical moment, the current moment and the future moment;
Acquiring a second group of track sequences of a target rear point corresponding to a second limb pair of the controlled object at the current moment, wherein each track sequence in the second group of track sequences comprises values of a parameter corresponding to the target rear point at the historical moment, the current moment and the future moment;
The limb pair of the controlled object comprises the first limb pair and the second limb pair, the target point comprises the target front point and the target rear point, and the track sequence information comprises the first group of track sequences and the second group of track sequences.
3. The method of claim 1, wherein the track sequence information comprises a set of track sequences, each track sequence in the set of track sequences comprising a value of a parameter corresponding to the target point at the set of historical moments, the current moment, and the set of future moments; the updating the motion state information according to the motion state reference information includes:
And updating the set of track sequences according to the first set of sub-track sequences, wherein a predicted value of a parameter corresponding to the target point in each sub-track sequence in the first set of sub-track sequences at the current time and the set of future times is used for updating a value of a parameter corresponding to the target point in each track sequence in the set of track sequences at the current time and the set of future times.
4. A method according to claim 3, wherein said updating said set of track sequences from said first set of sub-track sequences comprises at least one of:
Updating the values of the position parameters of the target points in the position track sequences in the group of track sequences at the current moment and the group of future moments according to the predicted values of the position parameters of the target points in the position track sequences in the first group of sub track sequences at the current moment and the group of future moments;
Updating the values of the direction parameters of the target points in the direction track sequences in the group of track sequences at the current moment and the group of future moments according to the predicted values of the direction parameters of the target points in the direction track sequences in the first group of sub track sequences at the current moment and the group of future moments;
and updating the values of the speed vector parameters of the target points in the speed vector track sequences in the group according to the predicted values of the speed vector parameters of the target points in the speed vector sub track sequences in the first group at the current time and the future time.
5. The method of claim 1, wherein updating the current set of joint parameter values based on the predicted set of joint parameter values comprises at least one of:
Updating the current value of the position parameter corresponding to the group of joints in the current joint parameter value set according to the predicted value of the position parameter corresponding to the group of joints in the predicted joint parameter value set;
Updating the current values of the posture vector parameters corresponding to the group of joints in the current joint parameter value set according to the predicted values of the posture vector parameters corresponding to the group of joints in the predicted joint parameter value set;
updating the current value of the speed vector parameter corresponding to the group of joints in the current joint parameter value set according to the predicted value of the speed vector parameter corresponding to the group of joints in the predicted joint parameter value set.
6. The method according to any one of claims 1 to 5, wherein generating a predicted action instruction of the controlled object at a next time based on the action state estimation information includes:
when motion control information input by a target input device and used for controlling the controlled object is not detected at the current moment, generating the predicted motion instruction of the controlled object at the next moment according to the motion state estimated information;
When motion control information input by the target input device and used for controlling the controlled object is detected at the current moment, updating the motion state estimated information of the controlled object according to the motion control information input by the target input device to obtain updated motion state estimated information of the controlled object; and generating the predicted action instruction of the controlled object at the next moment according to the updated action state predicted information.
7. The method according to claim 6, wherein the updating the motion state estimation information of the controlled object according to the motion manipulation information input by the target input device includes:
Determining a target gait vector and a target speed scalar triggered by the motion control information according to the key identification of a group of keys which are touched under the condition that the motion control information input by the target input device comprises the key identification of the group of keys which are touched, wherein the target gait vector comprises the values of a group of gait parameters of the controlled object, each gait parameter in the group of gait parameters represents one gait of the controlled object, and the target speed scalar represents the values of the speed scalar parameters of the controlled object;
When the motion state estimation information comprises a current gait sequence and a current speed scalar subsequence in the motion state information, respectively updating the current gait sequence and the current speed scalar subsequence in the motion state estimation information according to the target gait vector and the target speed scalar, wherein the current gait sequence comprises values of the set of gait parameters of the controlled object at the set of historical moments, the current moment and the set of future moments, and the current speed scalar subsequence comprises values of the speed scalar parameters of the target point at the current moment and the set of future moments.
8. The method of claim 7, wherein the determining the target gait vector and target speed scalar triggered by the motion manipulation information based on the key identifications of the set of keys comprises:
Setting a value of a jump parameter in the target gait vector to a first value, setting a value of a gait parameter other than the jump parameter in the target gait vector to a second value different from the first value, and setting the target speed scalar to a preset first speed value when the key identification of the set of keys comprises the key identification of the jump combination key or the key identification of the jump key, wherein the set of gait parameters comprises the jump parameter, and the first speed value is greater than 0;
Setting a value of a sitting parameter in the target gait vector to the first value, setting a value of a gait parameter in the target gait vector other than the sitting parameter to the second value different from the first value, and setting the target speed scalar to 0 when the key identification of the set of keys includes a key identification of a sitting combination key or a key identification of a sitting key, wherein the set of gait parameters includes the sitting parameter;
Setting the value of a down parameter in the target gait vector to the first value, setting the value of a gait parameter in the target gait vector other than the down parameter to the second value different from the first value, and setting the target speed scalar to 0 when the key identification of the group of keys comprises the key identification of the down combination key or the key identification of the down key, wherein the group of gait parameters comprises the down parameter;
Setting the value of a running parameter in the target gait vector to the first value, setting the value of a gait parameter in the target gait vector other than the running parameter to the second value different from the first value, and setting the target speed scalar to a preset second speed value when the key identification of the group of keys comprises the key identification of the running combination key or the key identification of the running key, wherein the group of gait parameters comprises the running parameter, and the second speed value is larger than 0;
when the key identification of the set of keys includes a key identification of a standing combination key or a key identification of a standing key, setting a value of a standing parameter in the target gait vector to the first value, setting a value of a gait parameter other than the standing parameter in the target gait vector to the second value different from the first value, and setting the target speed scalar to 0, wherein the set of gait parameters includes the standing parameter.
9. The method of claim 7, wherein updating the current gait sequence and the current speed scalar sub-sequence in the motion state prediction information according to the target gait vector and the target speed scalar, respectively, comprises:
Updating the values of the set of gait parameters of the controlled object in the current gait sequence at the current moment and the set of future moments according to the target gait vector;
And updating the values of the speed scalar parameters of the target points in the current speed scalar subsequence at the current time and the set of future time according to the target speed scalar.
10. The method according to claim 7, wherein the updating the motion state estimation information of the controlled object according to the motion manipulation information input by the target input device includes:
determining a target rotation angle of the controlled object according to the key identification of the group of keys;
determining a target fusion coefficient according to the target gait vector;
When the motion state estimated information comprises a current expected direction and a current expected speed vector in the motion state information, updating the current expected direction in the motion state estimated information according to the target rotation angle and the target fusion coefficient, and updating the current expected speed vector in the motion state estimated information according to the target speed scalar and the target fusion coefficient.
11. The method of claim 10, wherein the step of determining the position of the first electrode is performed,
The updating the current gait sequence and the current speed scalar subsequence in the motion state estimation information according to the target gait vector and the target speed scalar respectively includes: updating the values of the set of gait parameters of the controlled object in the current gait sequence at the current moment and the set of future moments according to the target gait vector; updating the values of the speed scalar parameters of the target point in the current speed scalar subsequence at the current time and the set of future time according to the target speed scalar, so as to obtain the updated current speed scalar subsequence;
The updating the motion state estimation information of the controlled object according to the motion control information input by the target input device comprises the following steps: and updating the value of the speed scalar parameter of the target point in the updated current speed scalar subsequence at the current time and the set of future times according to the target fusion coefficient and the second expected speed vector when the current expected speed vector comprises a first expected speed vector corresponding to the target point and the first expected speed vector is updated to be a second expected speed vector.
12. The method according to claim 10, wherein the updating the motion state estimation information of the controlled object according to the motion manipulation information input by the target input device includes:
And when the motion state estimated information comprises the current expected correction amount in the motion state information, updating the current expected correction amount in the motion state estimated information according to the target fusion coefficient.
13. The method of claim 10, wherein the track sequence information comprises a set of track sequences, each track sequence in the set of track sequences comprising a value of a parameter corresponding to the target point at the set of historical moments, the current moment, and the set of future moments; the updating the motion state estimation information of the controlled object according to the motion control information input by the target input device comprises the following steps:
When the motion state estimation information includes a second group of sub-track sequences and the current expected direction in the motion state estimation information is updated to a target expected direction and the current expected speed vector in the motion state estimation information is updated to a target expected speed vector, updating values of position parameters of the target point in a position sub-track sequence in the second group of sub-track sequences at the set of future times according to the target expected direction and the target expected speed vector, updating values of direction parameters of the target point in a direction sub-track sequence in the second group of sub-track sequences at the set of future times, and updating values of speed vector parameters of the target point in a speed vector sub-track sequence in the second group of sub-track sequences at the set of future times;
The second group of sub-track sequences are sequences obtained by updating the values of a parameter corresponding to the target point in each track sequence in the group of track sequences at the current moment and the group of future moments according to the action state reference information.
14. An action instruction generating apparatus, comprising:
The system comprises an acquisition unit, a control unit and a control unit, wherein the acquisition unit is used for acquiring action state information of limbs of a controlled object on a corresponding target point at the current moment, and the action state information comprises track sequence information of the target point at a set of historical moment, the current moment and a set of future moment;
The prediction unit is used for inputting the action state information into a target prediction neural network model to obtain output action state reference information, wherein the action state reference information comprises a first group of sub-track sequences, and each sub-track sequence in the first group of sub-track sequences comprises a predicted value of a parameter corresponding to the target point at the current moment and the future moment;
The updating unit is configured to update the motion state information according to the motion state reference information to obtain motion state estimated information of the target point at a next moment, and includes: updating the current joint parameter value set according to the predicted joint parameter value set when the motion state information comprises the current joint parameter value set and the motion state reference information comprises the predicted joint parameter value set, wherein the current joint parameter value set comprises the current value of a group of parameters corresponding to a group of joints of the controlled object, and the predicted joint parameter value set comprises the predicted value of the group of parameters corresponding to the group of joints;
And the generating unit is used for generating a predicted action instruction of the controlled object at the next moment based on the action state estimated information.
15. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the method of any one of claims 1 to 13.
16. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run performs the method of any one of claims 1 to 13.
17. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 13 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210118989.0A CN116617663B (en) | 2022-02-08 | 2022-02-08 | Action instruction generation method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210118989.0A CN116617663B (en) | 2022-02-08 | 2022-02-08 | Action instruction generation method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116617663A CN116617663A (en) | 2023-08-22 |
| CN116617663B true CN116617663B (en) | 2024-08-13 |
Family
ID=87625239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210118989.0A Active CN116617663B (en) | 2022-02-08 | 2022-02-08 | Action instruction generation method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116617663B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110930483A (en) * | 2019-11-20 | 2020-03-27 | 腾讯科技（深圳）有限公司 | Role control method, model training method and related device |
| CN111223168A (en) * | 2020-01-17 | 2020-06-02 | 腾讯科技（深圳）有限公司 | Target object control method and device, storage medium and computer equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112270734B (en) * | 2020-10-19 | 2024-01-26 | 北京大米科技有限公司 | Animation generation method, readable storage medium and electronic equipment |
| CN114821006B (en) * | 2022-06-23 | 2022-09-20 | 盾钰（上海）互联网科技有限公司 | Twin state detection method and system based on interactive indirect reasoning |
-
2022
- 2022-02-08 CN CN202210118989.0A patent/CN116617663B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110930483A (en) * | 2019-11-20 | 2020-03-27 | 腾讯科技（深圳）有限公司 | Role control method, model training method and related device |
| CN111223168A (en) * | 2020-01-17 | 2020-06-02 | 腾讯科技（深圳）有限公司 | Target object control method and device, storage medium and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116617663A (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110930483B (en) | Role control method, model training method and related device | |
| US11586276B2 (en) | Systems and methods for generating complementary data for visual display | |
| US11069113B2 (en) | Method and apparatus for creating a computer simulation of an actor | |
| JP7061238B2 (en) | Reinforcement learning to train characters using heterogeneous target animation data | |
| Laszlo et al. | Interactive control for physically-based animation | |
| Kulić et al. | Anthropomorphic movement analysis and synthesis: A survey of methods and applications | |
| CN101916324B (en) | System and method for dependency graph evaluation for animation | |
| US6057859A (en) | Limb coordination system for interactive computer animation of articulated characters with blended motion data | |
| KR101098834B1 (en) | Apparatus and method for generating motion based on dynamics | |
| Kenwright | Generating responsive life-like biped characters | |
| CN111773686A (en) | Animation generation method and device, storage medium and electronic device | |
| Ishigaki et al. | Performance-based control interface for character animation | |
| WO2023088041A1 (en) | Method and apparatus for generating walking animation of virtual character, and device and storage medium | |
| JP2011508290A (en) | Motion animation method and apparatus | |
| CN113706666A (en) | Animation data processing method, non-volatile storage medium, and electronic device | |
| US11816772B2 (en) | System for customizing in-game character animations by players | |
| EP0808484B1 (en) | Method and apparatus for controlling the movement of a virtual body | |
| WO1997040471A1 (en) | Limb coordination system for interactive computer animation of articulated characters with blended motion data | |
| CN116617663B (en) | Action instruction generation method and device, storage medium and electronic equipment | |
| Naderi et al. | A reinforcement learning approach to synthesizing climbing movements | |
| Laszlo et al. | Predictive feedback for interactive control of physics-based characters | |
| Berseth | Scalable deep reinforcement learning for physics-based motion control | |
| Cheema et al. | Discovering Fatigued Movements for Virtual Character Animation | |
| Van de Panne | Control for simulated human and animal motion | |
| US20230306667A1 (en) | Procedural animation engine and editor for sample-based design of stylized walking gaits for bipedal robots |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40091945 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |