CN118133845A - A fusion method, device, equipment and storage medium for multi-channel semantic understanding - Google Patents

Abstract

The application relates to a fusion method, a device, equipment and a storage medium for multi-channel semantic understanding. The method comprises the following steps: and constructing a semantic understanding fusion model. Acquiring a multi-channel semantic text as a training sample data set, wherein the training sample data set comprises: voice information, visual information, and text information. The semantic understanding fusion model carries out NSP task training on the training sample data set to obtain a semantic sequence to be marked, and carries out channel marking on the semantic sequence to be marked based on a transducer architecture to obtain a semantic text of the marking sequence. The tag sequence semantic text adopts a seq2seq model to generate a standard instruction at a decoding layer, and the standard instruction carries out text structuring on the tag sequence semantic text to obtain the standard semantic text. By adopting the method, the problems of instruction conflict, redundancy and missing complementation among the multichannel data can be solved.

Description

A fusion method, device, equipment and storage medium for multi-channel semantic understanding

Technical Field

The present application relates to the field of computer technology, and in particular to a fusion method, device, equipment and storage medium for multi-channel semantic understanding.

Background technique

Since Bolt et al. proposed the concept of multi-channel user interface in the 1980s, multi-channel user interface has been increasingly developed with its efficient and natural interaction mode, and a series of multi-channel interaction systems have been studied and developed, from Quickset that combines voice and pen gestures to multi-channel systems that combine voice and visual gaze, as well as multi-channel interaction for handheld mobile devices. An important task in these multi-channel interaction systems is the joint processing of information input from each channel. There are two main types of multi-channel systems to solve the joint processing of information input from multiple channels: one is information fusion at the feature layer (also called early fusion), and the other is information fusion at the semantic layer (also called late fusion). Feature fusion is generally considered to be more suitable for closely related and synchronized channels, such as language and lip movements, because these two input channels provide corresponding information for the same semantic unit. However, when each channel of the system provides information with insufficient content and time tags, such as voice and pen input, these two channels provide different but complementary information. At this time, semantic layer fusion is needed.

At present, for the above two systems, when each channel of the system provides information with insufficient content and time stamp or the information provided by each channel is of low density, feature fusion becomes extremely difficult. The channels in this scenario are mainly the fusion of voice, gloves and back clip channels. The correlation between channels is poor, and feature fusion is difficult. The usual multi-channel semantic understanding only fills the semantic features of each channel according to the position of the word slot. When all the elements of a channel's command cannot be fully represented by one channel, it is an incomplete semantic slot and needs to be supplemented by other channel semantics. However, when faced with command conflicts and command redundancy between multiple commands, the semantic slot-based approach usually introduces rules to disambiguate command conflicts and command redundancy; this strategy leads to too many rules and difficulty in implementation in complex scenarios.

Summary of the invention

Based on this, it is necessary to provide a multi-channel semantic understanding fusion method, device, equipment and storage medium that can efficiently and conveniently fuse multi-channel semantic texts in response to the above technical problems.

A fusion method for multi-channel semantic understanding, the method comprising:

Build a semantic understanding fusion model.

Multi-channel semantic text is obtained as a training sample data set, and the training sample data set includes: voice information, visual information and text information.

The semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample dataset, and then performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the semantic text of the labeled sequence.

The labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

In one of the embodiments, it also includes: using unsupervised data sampling technology and data universal annotation technology to build a semantic understanding fusion model.

In one of the embodiments, it also includes: identifying original instructions of multiple channels through a semantic understanding fusion model to obtain semantic texts corresponding to the original instructions as a training sample data set.

In one embodiment, the semantic understanding fusion model also includes: the semantic understanding fusion model obtains multi-channel semantic sequences to be labeled by performing NSP task training between any two consecutive sentences in the training sample data set. A semantic cross-fusion model is constructed based on the semantic sequences to be labeled. The semantic cross-fusion model uses segment_id to perform channel labeling on the semantic sequences to be labeled in each channel based on the Transformer architecture to obtain the labeled sequence semantic text.

In one embodiment, the semantic understanding fusion model includes a decoding layer, and also includes: extracting keyword groups through the decoding layer according to the semantic text of the tag sequence, masking the keyword groups as a dictionary to be decoded, and obtaining candidate decoding dictionaries, and the candidate decoding dictionaries generate original instructions from sequence to sequence.

In one of the embodiments, it also includes: the marked sequence semantic text uses a seq2seq model at the decoding layer to parse the original instruction to obtain the original instruction task, generates a standard instruction based on the original instruction task, and the standard instruction uses sequence annotation to perform text structuring on the marked sequence semantic text to obtain a standard semantic text.

In one embodiment, the method further includes: training a semantic understanding fusion model according to the multi-channel semantic text and the standard semantic text to obtain a trained semantic understanding fusion model. The multi-channel semantic understanding fusion is performed using the trained semantic understanding fusion model.

A fusion device for multi-channel semantic understanding, the device comprising:

The semantic understanding fusion model building module is used to build the semantic understanding fusion model.

The sample data set acquisition module is used to acquire multi-channel semantic text as a training sample data set. The training sample data set includes: voice information, visual information and text information.

The sequence semantic text tagging module is used for the semantic understanding fusion model. By training the training sample data set with the NSP task, the semantic sequence to be labeled is obtained. Based on the Transformer architecture, the channel of the semantic sequence to be labeled is annotated to obtain the labeled sequence semantic text.

The semantic understanding fusion module is used to generate standard instructions for the labeled sequence semantic text using the seq2seq model at the decoding layer. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Build a semantic understanding fusion model.

Multi-channel semantic text is obtained as a training sample data set, and the training sample data set includes: voice information, visual information and text information.

The semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample dataset, and then performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the semantic text of the labeled sequence.

The labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

Build a semantic understanding fusion model.

Multi-channel semantic text is obtained as a training sample data set, and the training sample data set includes: voice information, visual information and text information.

The semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample dataset, and then performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the semantic text of the labeled sequence.

The labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

The above-mentioned multi-channel semantic understanding fusion method, device, equipment and storage medium mark the text of different channels based on the pre-trained semantic understanding fusion model, improve the difference between channels, which is conducive to improving the semantic difference between different channels and improving the model's ability to express semantic features. Compared with multi-channel fusion based on the feature level, the number of data annotations and the difficulty of fusion can be reduced. In addition, the constrained decoding method is adopted in the semantic fusion stage, which can effectively improve the generation ability of annotation instructions and ensure the correctness and comprehensibility of instruction generation. With the help of the understanding ability of the pre-trained language model, redundant instructions and single-channel information missing instructions can be complemented, which helps to ensure accuracy and realize efficient and convenient fusion of multi-channel semantic text.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a fusion method for multi-channel semantic understanding in one embodiment;

FIG2 is a structural block diagram of a fusion device for multi-channel semantic understanding in one embodiment;

FIG. 3 is a diagram showing the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in FIG1 , a fusion method for multi-channel semantic understanding is provided, comprising the following steps:

Step 102, constructing a semantic understanding fusion model.

Step 104: Acquire multi-channel semantic text as a training sample data set.

The training sample data set includes: voice information, visual information and text information.

Step 106, the semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample data set, and performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the labeled sequence semantic text.

Specifically, there is currently an identifier marked as segment_id in the semantic understanding fusion model, which is used to distinguish different sentences or fragments. In the scenario of multi-sentence input, in order to enable the model to understand and process the relationship between multiple sentences, these sentences are appropriately marked. For models based on the Transformer architecture (such as BERT, GPT, etc.), each input sentence needs to add a special start tag ([CLS]) and end tag ([SEP]). When processing two sentences, two segment_ids can be used to distinguish different sentences. Specifically, set the identifier of the first sentence to 0 and the identifier of the second sentence to 1. Furthermore, the sequence of each channel uses segment_id as a marker for the semantic text of different channels to improve the difference between channel texts and the model's ability to understand texts.

Step 108, the labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions, and the standard instructions perform text structuring on the labeled sequence semantic text to obtain a standard semantic text.

Specifically, a text generation model based on constrained decoding seq2seq is used to implement the standard instruction generation task. At the decoding layer, the decoding dictionary is masked, and all words and special character sets contained in the input text are used as candidate words in the decoding dictionary:

;

;

in, To obtain the candidate decoding dictionary, mask_seq represents the mask value of each word in the dictionary, the sequence length is equal to the length of all dictionaries, dictAllEmb is the embedding corresponding to all words in the dictionary, and embedding is a vector.

Furthermore, other contents are constructed according to a sequence-to-sequence generation model to obtain a labeled sequence semantic text, and standard instructions perform text structuring on the labeled sequence semantic text to obtain a standard semantic text.

Furthermore, the semantic understanding fusion model is trained according to the multi-channel semantic text and the standard semantic text to obtain a trained semantic understanding fusion model. The multi-channel semantic understanding fusion is performed by using the trained semantic understanding fusion model.

In the above-mentioned multi-channel semantic understanding fusion method, the texts of different channels are marked based on the pre-trained semantic understanding fusion model to improve the differences between channels, which is conducive to improving the semantic differences between different channels and improving the model's ability to express semantic features. Compared with multi-channel fusion based on the feature level, the number of data annotations and the difficulty of fusion can be reduced. In addition, the constrained decoding method is used in the semantic fusion stage, which can effectively improve the ability to generate annotation instructions and ensure the correctness and comprehensibility of instruction generation. With the help of the understanding ability of the pre-trained language model, redundant instructions and single-channel information missing instructions can be complemented, which helps to ensure accuracy and realize efficient and convenient fusion of multi-channel semantic texts.

In one of the embodiments, unsupervised data sampling technology and data universal annotation technology are used to build a semantic understanding fusion model.

It is worth noting that the pre-trained semantic understanding fusion model is based on massive data, with the help of unsupervised data sampling technology and data back-annotation technology to perform multi-task model training, so as to express text sequences. In the model training process, the prediction of the MASK word of the original text and the judgment of whether the sentence pair of the original text is the upper and lower sentences are usually adopted. The pre-trained semantic understanding fusion model has a strong deep semantic understanding ability and can identify the wrong text input into the channel. Since the pre-trained semantic understanding fusion model has a good text feature expression ability, it can identify wrong semantic instructions and correct them. The semantic analysis provides accurate instruction data support. For example: the output instruction data of the voice channel is [a pair forward], which is corrected to [a team forward] after fusion by the pre-trained semantic understanding fusion model in this solution.

In one of the embodiments, a semantic understanding fusion model is used to identify original instructions of multiple channels and obtain semantic texts corresponding to the original instructions as a training sample data set.

In one embodiment, the semantic understanding fusion model obtains multi-channel semantic sequences to be labeled by performing NSP task training between any two consecutive sentences in the training sample data set. A semantic cross-fusion model is constructed based on the semantic sequences to be labeled. The semantic cross-fusion model uses segment_id to perform channel labeling on the semantic sequences to be labeled in each channel based on the Transformer architecture to obtain the labeled sequence semantic text.

It is worth noting that in the multi-channel scenario, each channel can be regarded as a sentence. The text of all channels needs to be used as a sequence input model. The semantic information of some channels is incomplete, and it is necessary to use a pre-trained language model to enhance the language understanding ability. The pre-trained semantic understanding fusion model usually does not directly mark different sentences, but uses the Next Sentence Prediction (NSP) task to train the model to model the relationship between two consecutive sentences. The NSP task aims to enable the model to determine whether two sentences are closely related or independent of each other. Therefore, it is necessary to mark the sentences of the input multi-channel text to further improve the semantic understanding ability of the text. It can be seen that marking the input channel input data improves the differences between the texts of each channel, which is conducive to learning the features between different texts.

In one embodiment, the semantic understanding fusion model includes a decoding layer. The decoding layer extracts key words according to the semantic text of the tag sequence, and performs masking on the key words as a dictionary to be decoded to obtain a candidate decoding dictionary, which generates original instructions from sequence to sequence.

In one of the embodiments, the marked sequence semantic text uses a seq2seq model at the decoding layer to parse the original instruction to obtain the original instruction task, and a standard instruction is generated based on the original instruction task. The standard instruction uses sequence labeling to perform text structuring on the marked sequence semantic text to obtain a standard semantic text.

It is worth noting that the fusion is mainly aimed at the voice input transcription channel, gesture semantic channel and wearable device action representation semantic channel. The three channels will have conflicts, redundancy and complementarity. For example: when the text of the three channels [voice: 1 group advances 100 meters], [gesture: advance], and [action: advance], there is redundancy in the semantics of the three channels. In order to accurately understand the semantic information from the three channels, semantic fusion is required. With the help of the semantic understanding ability of the generative pre-trained semantic understanding fusion model, the domain data is optimized and adjusted to generate standard instructions, providing standard instructions for semantic understanding. Multi-channel semantic fusion adopts a sequence generation method based on a generative pre-trained language model combined with constrained decoding seq2seq to achieve the generation of standard instructions. It can be seen that in order to fully retain the semantic information of the original text, the sequence generation method of constrained decoding is adopted in the instruction generation process to improve the ability to generate standard instructions during the fusion process.

In one embodiment, a semantic understanding fusion model is trained based on multi-channel semantic texts and standard semantic texts to obtain a trained semantic understanding fusion model. Multi-channel semantic understanding fusion is performed using the trained semantic understanding fusion model.

It is worth noting that the multi-channel semantic text obtains the fused standard instructions through the fusion model. The machine model cannot accurately understand the annotation instructions, and the standard instructions need to be converted into fields corresponding to the semantic understanding. In this scheme, attributes, objects and actions are used as semantic objects to carry out the design of semantic understanding of annotation instructions. For example, the instruction [quickly pursue the enemy in the north direction] is converted into a semantic instruction: [attribute: quick], [object: enemy in the north direction], [action: pursuit]. The semantic understanding of the annotation instruction is a task of annotation information extraction, which is implemented by the pre-trained language model + sub-classifier. Taking the above three-channel text [voice: 1 group forward 100 meters], [gesture: forward], [action: forward] as an example, the text of the three channels is converted into three sentences and formed into a sequence [voice: 1 group forward 100 meters; gesture: forward; action: forward], and the sequence is used as the input of the pre-trained language model.

It should be understood that, although the various steps in the flowchart of FIG. 1 are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in FIG. 1 may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

In one embodiment, as shown in FIG2 , a multi-channel semantic understanding fusion device is provided, including: a semantic understanding fusion model building module 202, a sample data set acquisition module 204, a sequence semantic text tagging module 206 and a semantic understanding fusion module 208, wherein:

The semantic understanding fusion model building module 202 is used to build a semantic understanding fusion model.

The sample data set acquisition module 204 is used to acquire multi-channel semantic text as a training sample data set. The training sample data set includes: voice information, visual information and text information.

The sequence semantic text labeling module 206 is used for the semantic understanding fusion model to obtain the semantic sequence to be labeled by performing NSP task training on the training sample data set, and to perform channel labeling on the semantic sequence to be labeled based on the Transformer architecture to obtain the labeled sequence semantic text.

The semantic understanding fusion module 208 is used to generate standard instructions for the marked sequence semantic text using a seq2seq model at the decoding layer. The standard instructions perform text structuring on the marked sequence semantic text to obtain a standard semantic text.

For the specific definition of a fusion device for multi-channel semantic understanding, please refer to the definition of a fusion method for multi-channel semantic understanding above, which will not be repeated here. Each module in the above-mentioned fusion device for multi-channel semantic understanding can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG3. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fusion method of multi-channel semantic understanding is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

Those skilled in the art will understand that the structures shown in FIGS. 2-3 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Build a semantic understanding fusion model.

Multi-channel semantic text is obtained as a training sample data set, and the training sample data set includes: voice information, visual information and text information.

The semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample dataset, and then performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the semantic text of the labeled sequence.

The labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

In one embodiment, when the processor executes the computer program, the following steps are further implemented: the semantic understanding fusion model obtains multi-channel semantic sequences to be marked by performing NSP task training between any two consecutive sentences in the training sample data set. A semantic cross-fusion model is constructed based on the semantic sequences to be marked. The semantic cross-fusion model uses segment_id to perform channel marking on the semantic sequences to be marked in each channel based on the Transformer architecture to obtain the marked sequence semantic text.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: the marked sequence semantic text is parsed using a seq2seq model at the decoding layer to obtain the original instruction task, a standard instruction is generated based on the original instruction task, and the standard instruction uses sequence annotation to perform text structuring on the marked sequence semantic text to obtain a standard semantic text.

In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Build a semantic understanding fusion model.

Multi-channel semantic text is obtained as a training sample data set, and the training sample data set includes: voice information, visual information and text information.

The semantic understanding fusion model obtains the semantic sequence to be labeled by performing NSP task training on the training sample dataset, and then performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain the semantic text of the labeled sequence.

The labeled sequence semantic text uses a seq2seq model in the decoding layer to generate standard instructions. The standard instructions perform text structuring on the labeled sequence semantic text to obtain standard semantic text.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the semantic understanding fusion model obtains multi-channel semantic sequences to be marked by performing NSP task training between any two consecutive sentences in the training sample data set. A semantic cross-fusion model is constructed based on the semantic sequences to be marked. The semantic cross-fusion model uses segment_id to perform channel marking on the semantic sequences to be marked in each channel based on the Transformer architecture to obtain the marked sequence semantic text.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the marked sequence semantic text is parsed using a seq2seq model at the decoding layer to obtain the original instruction task, and a standard instruction is generated based on the original instruction task. The standard instruction uses sequence annotation to perform text structuring on the marked sequence semantic text to obtain a standard semantic text.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the invention. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements may be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (10)

1. A fusion method for multi-channel semantic understanding, characterized in that the method comprises: Build a semantic understanding fusion model; Acquire multi-channel semantic text as a training sample data set, wherein the training sample data set includes: voice information, visual information and text information; The semantic understanding fusion model performs NSP task training on the training sample data set to obtain a semantic sequence to be labeled, and performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain a labeled sequence semantic text; The labeled sequence semantic text uses a seq2seq model to generate standard instructions at the decoding layer, and the standard instructions perform text structuring on the labeled sequence semantic text to obtain a standard semantic text. 2. The method according to claim 1, characterized in that constructing a semantic understanding fusion model comprises: Unsupervised data sampling technology and data universal annotation technology are used to build a semantic understanding fusion model. 3. The method according to claim 1, characterized in that obtaining multi-channel semantic text as a training sample data set comprises: The semantic understanding fusion model is used to identify original instructions of multiple channels and obtain semantic texts corresponding to the original instructions as a training sample data set. 4. The method according to claim 3 is characterized in that the semantic understanding fusion model obtains a semantic sequence to be labeled by performing NSP task training on the training sample data set, and performs channel annotation on the semantic sequence to be labeled based on the Transformer architecture to obtain a labeled sequence semantic text, including: The semantic understanding fusion model obtains a multi-channel semantic sequence to be marked by performing NSP task training between any two consecutive sentences in the training sample data set; A semantic cross-fusion model is constructed according to the semantic sequence to be marked. The semantic cross-fusion model uses segment_id to perform channel marking on the semantic sequence to be marked in each channel based on the Transformer architecture to obtain a marked sequence semantic text. 5. The method according to any one of claims 1 to 4, characterized in that the semantic understanding fusion model includes a decoding layer; Before the step of generating a standard instruction by using a seq2seq model at a decoding layer for the semantic text of the marked sequence, and the standard instruction performs text structuring on the semantic text of the marked sequence to obtain a standard semantic text, the method further includes: According to the semantic text of the tag sequence, a keyword group is extracted through the decoding layer, and the keyword group is used as a dictionary to be decoded for masking to obtain a candidate decoding dictionary, and the candidate decoding dictionary generates original instructions from sequence to sequence. 6. The method according to claim 5 is characterized in that the tag sequence semantic text uses a seq2seq model at a decoding layer to generate standard instructions, and the standard instructions perform text structuring on the tag sequence semantic text to obtain a standard semantic text, including: The labeled sequence semantic text uses a seq2seq model at the decoding layer to parse the original instruction to obtain an original instruction task, and generates a standard instruction based on the original instruction task. The standard instruction uses sequence annotation to perform text structuring on the labeled sequence semantic text to obtain a standard semantic text. 7. The method according to claim 6 is characterized in that after the step of generating standard instructions by using a seq2seq model at a decoding layer of the labeled sequence semantic text, and the standard instructions structuring the labeled sequence semantic text to obtain a standard semantic text, the method further comprises: According to the multi-channel semantic text and the standard semantic text, the semantic understanding fusion model is trained to obtain a trained semantic understanding fusion model; Multi-channel semantic understanding fusion is performed through the trained semantic understanding fusion model. 8. A fusion device for multi-channel semantic understanding, characterized in that the device comprises: Semantic understanding fusion model building module, used to build a semantic understanding fusion model; A sample data set acquisition module is used to acquire multi-channel semantic text as a training sample data set, wherein the training sample data set includes: voice information, visual information and text information; A sequence semantic text labeling module is used for the semantic understanding fusion model to obtain a semantic sequence to be labeled by performing NSP task training on the training sample data set, and to label the semantic sequence to be labeled based on the Transformer architecture to obtain a labeled sequence semantic text; The semantic understanding fusion module is used for generating standard instructions for the semantic text of the tag sequence by using a seq2seq model at a decoding layer. The standard instructions perform text structuring on the semantic text of the tag sequence to obtain a standard semantic text. 9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program. 10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.