CN118350416A - Multimodal semantic communication method, system, device and medium based on large model - Google Patents

Abstract

The application provides a multi-mode semantic communication method, a system, equipment and a medium based on a large model, which relate to the technical field of semantic communication and acquire a pre-trained large language model codec and multi-mode data; constructing a receiving and transmitting end semantic big model to be trained in steps by utilizing a coder-decoder, a projector and a pre-trained big language model coder-decoder which are related to a mode to be trained; maintaining parameters of a pre-trained large language model coder-decoder unchanged, and circularly and alternately training the coder-decoder, the projector and the channel coder-decoder to obtain a receiving-transmitting end semantic large model; and transmitting and reconstructing multi-mode data by using the semantic big model of the receiving and transmitting end. By the method, unified semantic extraction and fusion characterization of the multi-mode data are realized, and the accuracy and stability of semantic mapping are ensured, so that information redundancy among modes is eliminated, efficient multi-mode data semantic alignment and fusion are realized, and the universality and expansibility of a communication system are improved.

Description

Multimodal semantic communication method, system, device and medium based on large model

Technical Field

The present application relates to the field of semantic communication technology, and in particular to a multimodal semantic communication method, system, device and medium based on a large model.

Background technique

Existing semantic communication solutions are mainly based on deep learning technology, which aims to use neural networks to accurately extract meaning or features from source data and transmit the required semantic information in a highly simplified way. In practical applications, multimodal data can often provide more comprehensive and richer information, which helps to more accurately understand the user's intentions and needs.

However, most existing semantic communication solutions only consider unimodal scenarios, and extract and represent semantics by designing specialized models, that is, for specific tasks such as text processing, image recognition or speech recognition. Although this approach can achieve good results on specific tasks, it faces problems such as insufficient semantic extraction, lack of versatility and scalability when processing multimodal data.

At present, most multimodal semantic communication solutions adopt the method of multimodal fusion, that is, fusing data from different modalities, and then using deep learning and other technologies for semantic extraction and representation. However, these solutions still have the following problems: Due to the differences and complementarities between data of different modalities, how to effectively fuse this information and extract accurate semantic information is a key issue. Existing multimodal fusion methods often fail to fully utilize the correlation and complementarity between different modalities, resulting in redundant or inaccurate extracted semantic information.

Therefore, in view of the shortcomings of existing semantic communication solutions in multimodal scenarios, it is necessary to explore new semantic communication solutions to achieve unified semantic information extraction and representation of multimodal data and improve the accuracy and efficiency of semantic communication.

Summary of the invention

The present application provides a large-model-based multimodal semantic communication method, system, device and medium to solve the problem of semantic alignment and fusion transmission between multimodal data in the semantic communication method.

In a first aspect of an embodiment of the present application, a multimodal semantic communication method based on a large model is proposed, the method comprising:

Obtain a pre-trained large language model codec and multimodal data including text, image, audio, and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus;

Using the modality-related codec and the modality-related projector to be trained, and the pre-trained large language model codec, construct a semantic codec based on the large model to be trained, and using the channel codec to be trained and the semantic codec based on the large model to be trained, construct a semantic large model of the transceiver to be trained;

Under the premise of keeping the parameters of the pre-trained large language model codec unchanged, cyclically and alternately training the codec and projector related to the modality to be trained, and the channel codec to be trained, to obtain a large semantic model of the transceiver;

The multimodal data is transmitted and reconstructed using the semantic big model of the transmitting and receiving ends.

The method specifically includes:

The transmitter jointly encodes the multimodal data from the source through the semantic encoder based on the large model in the semantic large model of the transmitter and receiver, captures and fuses the semantic associations between different modalities, and generates a high-dimensional semantic space representation shared across modalities;

The transmitter converts the cross-modal shared high-dimensional semantic space representation into semantic symbols through an adaptive channel encoder in the semantic large model of the transmitter and receiver, selects a coding rate matching the channel state for encoding, and selects a transmission strategy matching the channel state for transmission;

The receiving end restores the received semantic symbols into a high-dimensional semantic space representation shared across modalities through an adaptive channel decoder in the semantic large model of the transceiver end, selects a decoding rate matching the channel state for decoding, and selects an error correction strategy matching the channel state for error correction;

The receiving end reconstructs the restored cross-modal shared high-dimensional semantic space representation through a semantic decoder based on the large model in the semantic large model of the transmitting and receiving end to obtain reconstructed multimodal data.

The training process of the large semantic model of the transmitting and receiving end is as follows:

The first stage of training: freezing the parameters of the pre-trained large language model codec, and separately training the channel codec to be trained to adapt to the channel state;

The second stage of training: freezing the parameters of the channel codec trained to a certain degree, and training the parameters of the codec and projector related to the modality to be trained;

The first stage training and the second stage training are cyclically alternated to optimize the performance of the channel codec to be trained, the modality-related codec to be trained, and the modality-related projector, until the loss function value of the semantic large model of the transceiver converges or meets the predetermined performance index.

The training process of the large semantic model of the transmitting and receiving end further includes:

During the training process, a difference between a reconstructed communication intent of the reconstructed multimodal data and an actual communication intent of the multimodal data from the source is calculated;

According to the difference, evaluating whether the semantic big model of the sending and receiving end maintains semantic consistency;

Based on the evaluation results, the parameters and training strategy of the semantic large model of the transmitting and receiving ends are adjusted to reduce the difference.

Before jointly encoding the multimodal data from the information source, the method includes:

Preprocessing the multimodal data so that the preprocessed multimodal data has a preset data format;

The joint encoding of the multimodal data from the information source is implemented by the semantic encoder based on the large model, and the semantic encoder based on the large model includes a modality encoder, an input projector and a large language model encoder;

The modal encoder is used to extract features from the preprocessed multimodal data;

The input projector is used to align the extracted features so that data from different modalities can be effectively integrated in the same feature space;

The large language model encoder is used to map data of different modalities into a unified semantic space to achieve semantic information fusion of multimodal data.

The large model-based semantic decoder includes a modality decoder, an output projector, and a large language model decoder. The large model-based semantic decoder reconstructs the restored cross-modal shared high-dimensional semantic space representation to obtain reconstructed multimodal data, including:

The large language model decoder is used to restore the high-dimensional semantic space representation from a unified semantic space into a first feature of different modal data;

The output projector is used to decode the first feature of the different modal data to obtain the second feature of the different modal data;

The modality decoder is used to reconstruct the second features of the different modality data into the original multi-modality data.

After the multimodal data is transmitted and reconstructed using the large semantic model of the transceiver, the method includes:

Based on the multimodal data reconstructed by the large semantic model of the transmitting and receiving end, a downstream task execution function component is used to execute corresponding downstream tasks to obtain output results;

Optimizing and adjusting the parameters of the downstream task execution functional component based on the output result to obtain an adjusted downstream task execution functional component;

The adjusted downstream task execution functional component executes the corresponding task based on the user's interactive instructions and the reconstructed multimodal data.

In a second aspect of an embodiment of the present application, a multimodal semantic communication system based on a large model is proposed, the system comprising:

An acquisition module is used to acquire a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus;

A construction module is used to construct a semantic codec based on a large model to be trained by using the codec and projector related to the modality to be trained, and the pre-trained large language model codec, and to construct a semantic large model of the transceiver to be trained by using the channel codec to be trained and the semantic codec based on the large model to be trained;

A training module is used to cyclically and alternately train the codec and projector related to the modality to be trained, and the channel codec to be trained, under the premise of keeping the parameters of the pre-trained large language model codec unchanged, to obtain a large semantic model of the transceiver;

The transmission and reconstruction module is used to transmit and reconstruct the multimodal data by using the semantic big model of the transmitting and receiving end.

In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the large model-based multimodal semantic communication method described in any one of the first aspects above.

In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program/instruction is stored. When the computer program/instruction is executed by a processor, the multimodal semantic communication method based on a large model as described in any one of the first aspects above is implemented.

The present application includes the following advantages: The present application provides a multimodal semantic communication method, system, device and medium based on a large model, obtains a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus; uses the modality-related codec and the modality-related projector to be trained, and the pre-trained large language model codec to construct a semantic codec based on a large model to be trained, and uses the channel codec to be trained and the semantic codec based on the large model to be trained to construct a large semantic model of the transceiver to be trained; on the premise of keeping the parameters of the pre-trained large language model codec unchanged, the modality-related codec and the modality-related projector to be trained, and the channel codec to be trained are cyclically and alternately trained to obtain a large semantic model of the transceiver; uses the large semantic model of the transceiver to transmit and reconstruct the multimodal data. The above method can realize unified semantic extraction and fusion representation of multimodal data, ensure the accuracy and stability of semantic mapping, eliminate information redundancy between modalities, realize efficient semantic alignment and fusion of multimodal data, and improve the versatility and scalability of communication systems.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

FIG1 is a flowchart of a multimodal semantic communication method based on a large model provided in an embodiment of the present application;

FIG2 is a flow chart of a multimodal semantic communication method based on a large model provided in an embodiment of the present application;

FIG3 is a schematic diagram of a training process of a large semantic model of a transmitting and receiving end provided in an embodiment of the present application;

FIG4 is an architecture diagram of a multimodal semantic communication system based on a large model provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of an electronic device provided in an embodiment of the present application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

Existing semantic communication solutions are mainly based on deep learning technology, which aims to use neural networks to accurately extract meaning or features from source data and transmit the required semantic information in a highly simplified way. The core of this technology is to extract key semantic information from massive data and transmit semantic information in a highly optimized way, thereby greatly saving bandwidth resources and improving network capacity and communication efficiency. Among them, multimodal data refers to a data set composed of different modes (such as text, images, audio, video, etc.). In practical applications, multimodal data can often provide more comprehensive and richer information, which helps to more accurately understand the user's intentions and needs.

However, most existing semantic communication solutions only consider unimodal scenarios, and perform semantic extraction and representation by designing specialized models, that is, targeting specific tasks such as text processing, image recognition, or speech recognition. Although this approach can achieve good results on specific tasks, it faces problems such as insufficient semantic extraction, lack of versatility and scalability when processing multimodal data. Specifically, due to the differences in the expression of data in different modalities, unimodal semantic communication may not be able to fully capture and transmit all the semantic information of the original data. Although unimodal semantic communication is relatively simple in model design and processing flow, complex pre-processing and post-processing steps may still be required in certain specific tasks or scenarios. When it is necessary to process information in multiple modalities, the model may need to be redesigned or adjusted, which increases the difficulty of development and maintenance.

At present, most multimodal semantic communication solutions adopt the method of multimodal fusion, that is, fusing data from different modalities, and then using deep learning and other technologies for semantic extraction and representation. However, these solutions still have the following problems: Due to the differences and complementarities between data of different modalities, how to effectively fuse this information and extract accurate semantic information is a key issue. Existing multimodal fusion methods often fail to fully utilize the correlation and complementarity between different modalities, resulting in redundant or inaccurate extracted semantic information. There are differences in the representation of different modal data. For example, text data is usually represented by word vectors, while image data is represented by pixel values or feature maps. This makes it difficult to fuse and extract semantic information from multimodal data, and it is difficult to form a unified representation method.

The reasons for the above problems are: (1) The complexity of multimodal data: Multimodal data consists of data from different sources and in different representations. Its complexity and diversity make it difficult to extract and represent its semantics in a unified way. (2) The limitations of technological development: Although technologies such as deep learning have made significant progress in processing single-modal data, they still face many challenges in processing multimodal data. Current technologies cannot fully utilize the correlation and complementarity between different modalities to achieve accurate semantic extraction and representation.

Based on this, this application proposes a multimodal semantic communication method based on a large model, which aims to address the shortcomings of existing semantic communication solutions in multimodal scenarios, so as to achieve unified semantic information extraction and representation of multimodal data and improve the accuracy and efficiency of semantic communication.

In a first aspect of an embodiment of the present application, a multimodal semantic communication method based on a large model is provided. Referring to FIG. 1 , FIG. 1 is a flowchart of a multimodal semantic communication method based on a large model provided by an embodiment of the present application. The method comprises the following steps:

Step 101: obtaining a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus;

Step 102: constructing a semantic codec based on a large model to be trained by using the codec and projector related to the modality to be trained, and the pre-trained large language model codec, and constructing a semantic large model of the transceiver to be trained by using the channel codec to be trained and the semantic codec based on the large model to be trained;

Step 103: Under the premise of keeping the parameters of the pre-trained large language model codec unchanged, cyclically and alternately train the codec and projector related to the modality to be trained, and the channel codec to be trained to obtain a large semantic model of the transceiver;

Step 104: Utilize the large semantic model of the transmitting and receiving end to transmit and reconstruct the multimodal data.

In the embodiments of the present application, a large model refers to a machine learning model with large-scale parameters and complex computing structures. These models are usually built from deep neural networks and contain billions or even hundreds of billions of parameters. The large model aims to improve the expressiveness and predictive performance of the model, enabling it to handle more complex tasks and data. The large model learns complex patterns and features by training massive data and has a stronger generalization ability. A large language model (LLM) is a large model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks, such as text summarization, question answering, translation, etc. Using multi-layer neural networks to model the statistical laws and potential semantic information of language, the large language model will learn and abstract a large amount of text data to generate logical and coherent language output.

Multimodal data means that different forms of existence or sources of information can be called a modality. Therefore, data composed of two or more modalities is multimodal data, which can include text, pictures, audio, video, and mixed data.

Semantic communication is a communication method whose core idea is to extract the semantic information of the message to be sent at the source, rather than the bit stream of the transmitted message. This means that semantic communication focuses not only on the specific content of the message, but also on the meaning or characteristics behind the message. Semantic communication can pre-understand the business needs and environment, understand, extract and transmit the semantic features of the source, and ensure that the destination can understand the semantic features of the received source, so as to successfully recover the semantic-based source information. Compared with traditional grammatical communication, semantic communication does not require the decoding sequence to strictly match the encoding sequence, but only requires that the semantic information recovered by the receiving end matches the sent semantic information. This method helps to reduce the transmission bandwidth requirements of large-bandwidth services, improve communication efficiency, and improve user experience.

In order to clearly illustrate a multimodal semantic communication method based on a large model provided in an embodiment of the present application, a detailed description is given below in conjunction with Figure 2, which is a flow chart of a multimodal semantic communication method based on a large model provided in an embodiment of the present application.

When implementing step 101, the core of the multimodal semantic communication method based on a large model proposed in this application is the pre-trained large language model codec, which is pre-trained through a corpus so that the pre-trained large language model codec has good language understanding and generation capabilities. At the same time, multimodal data including text, images, audio and video are obtained for semantic communication.

When step 102 is specifically implemented, a semantic codec based on a large model to be trained is constructed by using a codec to be trained and a projector to be trained and a pre-trained large language model codec, and a semantic large model of the transceiver to be trained is constructed by using a channel codec to be trained and the constructed semantic codec based on a large model to be trained. The codecs to be trained and the modality-related codecs include text codecs, image codecs, audio codecs, and video codecs, and the projectors to be trained and the modality-related codecs include image input/output projectors, audio input/output projectors, and video input/output projectors.

When step 103 is specifically implemented, under the premise of keeping the parameters of the pre-trained large language model codec unchanged, the channel codec to be trained, the codec related to the modality and the projector related to the modality are trained alternately in a loop until the training is completed to obtain a large semantic model of the transceiver.

When step 104 is specifically implemented, after the semantic big model at the transmitting and receiving end is trained, it is used to understand and represent the semantic information of data of different modalities, and realize the fusion, conversion, transmission and reconstruction of multimodal data.

In an optional embodiment of the present application, multimodal data is transmitted to a receiving end through a transmitting end, wherein, at the transmitting end, the semantic large model of the transceiver includes at least a semantic-channel joint encoding module, the semantic-channel joint encoding module includes a semantic encoder based on a large model and an adaptive channel encoder, and at the receiving end, the semantic large model of the transceiver includes at least a semantic-channel joint decoding module, the semantic-channel joint decoding module includes an adaptive channel decoder and a semantic decoder based on a large model. The method specifically includes: at the transmitting end, the multimodal data from the source is jointly encoded by a semantic encoder based on a large model in the semantic large model of the transceiver, the semantic associations between different modalities are captured and integrated, and a high-dimensional semantic space representation shared across modalities is generated, and then the high-dimensional semantic space representation shared across modalities is converted into a semantic symbol through an adaptive channel encoder in the semantic large model of the transceiver, a coding rate matching the channel state is selected for encoding, and a transmission strategy matching the channel state is selected for transmission of semantic symbols. At the receiving end, the received semantic symbols are restored into a high-dimensional semantic space representation shared across modalities through the adaptive channel decoder in the semantic big model of the transceiver. The decoding rate matching the channel state is selected for decoding, and the error correction strategy matching the channel state is selected for error correction to ensure the accuracy and integrity of the decoded multimodal data. Finally, the restored high-dimensional semantic space representation shared across modalities is reconstructed through the big model-based semantic decoder in the semantic big model of the transceiver to obtain the reconstructed multimodal data.

In an optional embodiment of the present application, before the multimodal data from the information source is jointly encoded, the acquired multimodal data is preprocessed so that the preprocessed multimodal data has a preset data format, wherein the multimodal data can be text, image, audio, video, etc. Specifically, the acquired multimodal data is converted, cleaned, standardized, and feature extracted in a unified preset format to remove noise and irrelevant information. Through deep learning methods, the original unstructured multimodal data is converted into a form that can be understood and processed by a machine, such as word segmentation and part-of-speech tagging of text data, scaling, cropping, normalization, and conversion of image data into visual feature vectors, sampling, quantization, feature extraction, and conversion of audio data into acoustic feature sequences, etc.

In an optional embodiment of the present application, after preprocessing the multimodal data, a semantic encoder based on a large model is used to jointly encode the preprocessed multimodal features, capture and fuse the intrinsic semantic associations between different modalities, and generate a high-dimensional semantic space representation shared across modalities. The above-mentioned semantic encoder based on the large model includes a modality encoder, an input projector, and a large language model encoder; wherein the modality encoder extracts features from the preprocessed multimodal data, for example, the modality encoder can be a text encoder, an image encoder, an audio encoder, a video encoder, etc.; the input projector aligns the extracted features so that data of different modalities can be effectively integrated in the same feature space, for example, the input projector can be an image input projector, an audio input projector, a video input projector, etc.; the large language model encoder (corresponding to the LLM encoder in Figure 2) maps data of different modalities into a unified semantic space to achieve semantic information fusion of multimodal data (see the semantic large model encoder part in Figure 2). For example, when the data after multimodal data preprocessing is text, image, audio and video, after being encoded by the text encoder, image encoder, audio encoder and video encoder respectively, except for the text features, the remaining features corresponding to the image, audio and video are projected to the same feature space through the image input projector, audio input projector and video input projector respectively, and then mapped to a unified semantic space through the LLM encoder to realize the semantic information fusion of multimodal data.

In an optional embodiment of the present application, the semantic decoder based on the large model at the receiving end includes a modality decoder, an output projector and a large language model decoder. The semantic decoder based on the large model reconstructs the restored high-dimensional semantic space representation shared across modalities to obtain reconstructed multimodal data, specifically including: restoring the high-dimensional semantic space representation from a unified semantic space to the first feature of different modal data through a large language model decoder (corresponding to the LLM decoder in Figure 2); the output projector decodes the first feature of the different modal data to obtain the second feature of the different modal data, wherein the output projector can be an image output projector, an audio output projector, a video output projector, etc.; the modality decoder reconstructs the second feature of the different modal data into the original multimodal data, wherein the modality decoder can be a text decoder, an image decoder, an audio decoder, a video decoder, etc. (see the semantic large model decoder part in Figure 2). The joint semantic space is understood and reconstructed by the semantic decoder based on the large model, so that the receiving end can accurately understand the communication intention of the sending end. For example, the LLM decoder recovers the high-dimensional semantic space representation of multimodal data including text, images, audio and video from a unified semantic space into text features, image features, audio features and video features, and then the image features, audio features and video features are decoded into features of the corresponding modalities through the image output projector, audio output projector and video output projector respectively, and then reconstructed into image data, audio data and video data through the image decoder, audio decoder and video decoder respectively, among which the text features are directly reconstructed into text data through the text decoder.

In an optional embodiment of the present application, after using the above-mentioned semantic big model of the transceiver to transmit and reconstruct the multimodal data, the reconstructed multimodal data can also be used for downstream task execution (see the downstream task part in Figure 2). Specifically, based on the multimodal data reconstructed by the semantic big model of the transceiver, the downstream task execution functional component is used to execute the corresponding downstream task to obtain an output result; based on the output result, the parameters of the downstream task execution functional component are optimized and adjusted to obtain an adjusted downstream task execution functional component; and the adjusted downstream task execution functional component is used to execute the corresponding task based on the user's interactive instructions and the reconstructed multimodal data.

The downstream tasks performed by the functional components for performing downstream tasks described above may include various application scenarios, such as data recovery, video conferencing, smart home, virtual reality, etc. Through interaction and instructions with users, as well as decoded multimodal data, functions such as intelligent question and answer, intelligent control, and virtual interaction are realized. It can also be optimized and adjusted based on user feedback and system performance to provide more personalized and efficient services. In the downstream task execution stage, appropriate algorithms and models can be selected according to specific task requirements. For example, in question and answer tasks, a deep learning model based on an attention mechanism can be used to capture the association between questions and answers; in sentiment analysis tasks, sentiment dictionaries and machine learning algorithms can be used to identify sentiment tendencies in texts.

The training process of the semantic big model at the transceiver end proposed in the embodiment of the present application is completed through the collaboration of multiple modules, including a multimodal data acquisition module, a multimodal data preprocessing module, a semantic big model construction module at the transceiver end, a semantic big model training module at the transceiver end, a semantic big model output module and a semantic big model performance evaluation module. Refer to Figure 3, which is a schematic diagram of the training process of the semantic big model at the transceiver end provided in the embodiment of the present application. Next, the specific content of each of the above modules during the training process is described.

The multimodal data acquisition module is used to obtain diverse multimodal data from the real world or simulated environment, including text documents, voice conversation records, image files, videos and other types of data resources. For each modality, the sender-receiver semantic big model must ensure the diversity, richness and real-time nature of the data, so that it can cover all possible semantics and situations in the subsequent training stage, so that the sender-receiver semantic big model to be trained can better learn and understand the associations and differences between different modalities.

The multimodal data preprocessing module is used to perform basic processing and feature extraction on the voice, image and text data collected by the multimodal data acquisition module. For example, image information is classified and feature extracted to identify the key elements in the image; text information is processed by word segmentation and part-of-speech tagging to extract the key information in the text. At the same time, the multimodal data preprocessing module can also be used to identify the emotional features of multimodal data and extract emotional features. After the preprocessing is completed, the performance of each feature is numerically quantified to provide a unified data format for subsequent model training.

Furthermore, the semantic big model of the transceiver to be trained is constructed through the transceiver semantic big model construction module, and the semantic codec capable of processing multimodal data is constructed by using the modal codec, input/output projector and large language model codec. This module needs to be able to understand and represent the semantic information of different modal data to achieve the fusion and conversion of multimodal data. The semantic encoder based on the big model is used to map the multimodal input to a common high-dimensional semantic space; while the semantic decoder based on the big model is responsible for decoding the representation in this space back to the corresponding modal output. At the same time, this module also needs to build a channel codec to optimize the information transmission quality in a communication environment with noise and signal distortion. Through training, the channel codec has the ability to select a coding rate matching the channel state for encoding, and select a transmission strategy matching the channel state for the transmission of semantic symbols, so that the channel decoder has the ability to select a decoding rate matching the channel state for decoding, and select an error correction strategy matching the channel state for error correction.

After the semantic big model of the transceiver to be trained is constructed, the model is trained by the semantic big model training module of the transceiver based on iterative optimization and step-by-step training. The training process is as follows: In the first stage of training, the parameters of the pre-trained big language model codec are frozen, and the channel codec to be trained is trained separately to adapt to the channel state and optimize its anti-noise ability; in the second stage of training, when the channel codec to be trained is trained to a certain extent to dynamically adjust the transmission strategy according to the channel state, the parameters of the channel codec trained to a certain extent are frozen, and the parameters of the modality-related codec and the modality-related projector to be trained are trained to improve the semantic understanding and reconstruction capabilities of multimodal data. The first stage training and the second stage training are alternately cycled to optimize the performance of the channel codec to be trained, the modality-related codec to be trained, and the modality-related projector until the loss function value of the semantic big model of the transceiver converges or meets the predetermined performance indicators.

In the process of training the semantic large model of the transmitter and receiver, the parameters of the pre-trained large language model codec are always frozen. This is because the large language model codec has been pre-trained on the corpus and has good language understanding and generation capabilities. By freezing the parameters of the large language model codec, it can be ensured that its original capabilities are not changed during the training process, while focusing on training the codec and input/output projectors related to the modality to meet the communication needs of multimodal data. Through step-by-step training and iterative optimization, the model can achieve better performance in a shorter time and improve the efficiency of model training. Keep the parameters of the pre-trained large language model codec unchanged, and only train the codec related to the modality and the projector related to the modality. This ensures that the powerful language processing capabilities of the large language model codec are not destroyed, while enabling the model to better adapt to the semantic extraction and processing needs of multimodal data.

In the semantic big model output module, after receiving the multimodal data input, the semantic big model encodes the multimodal information into a unified semantic representation through the semantic encoder, and sends it out after adapting to the wireless channel through the channel encoder. After receiving the signal, the receiving end recovers the original semantic information through the channel decoder, and then the semantic decoder restores it to the expected multimodal data or generates the corresponding downstream task output.

During the training process of the semantic big model at the transceiver end, the performance of the semantic big model at the transceiver end during the training process is evaluated through the semantic big model performance evaluation module. Specifically, during the training process of the semantic big model at the transceiver end, the difference between the reconstructed communication intention of the reconstructed multimodal data and the actual communication intention of the multimodal data from the information source is calculated; based on the calculated difference, whether the semantic big model at the transceiver end maintains semantic consistency is evaluated; the evaluation result is fed back to the semantic big model training module at the transceiver end to guide further optimization of the semantic big model at the transceiver end, and the parameters and training strategies of the semantic big model at the transceiver end are adjusted to reduce the difference between the reconstructed communication intention of the reconstructed multimodal data and the actual communication intention of the multimodal data from the information source, thereby improving its overall performance.

The embodiment of the present application provides a multimodal semantic communication method based on a large model, which obtains a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus; a codec related to the modality to be trained and a projector related to the modality are used, as well as the pre-trained large language model codec, to construct a semantic codec based on a large model to be trained, and a channel codec to be trained and the semantic codec based on the large model to be trained are used to construct a semantic large model of the transceiver to be trained; on the premise of keeping the parameters of the pre-trained large language model codec unchanged, the codec related to the modality to be trained and the projector related to the modality, as well as the channel codec to be trained are cyclically and alternately trained to obtain a semantic large model of the transceiver; and the multimodal data is transmitted and reconstructed using the semantic large model of the transceiver. The above method can realize unified semantic extraction and fusion representation of multimodal data, ensure the accuracy and stability of semantic mapping, eliminate information redundancy between modalities, realize efficient semantic alignment and fusion of multimodal data, and improve the versatility and scalability of communication systems.

In a second aspect of an embodiment of the present application, a multimodal semantic communication system based on a large model is provided. Referring to FIG4 , FIG4 is an architecture diagram of a multimodal semantic communication system based on a large model provided in an embodiment of the present application, and the system includes:

An acquisition module 401 is used to acquire a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus;

A construction module 402 is used to construct a semantic codec based on a large model to be trained by using the codec and projector related to the modality to be trained, and the pre-trained large language model codec, and to construct a semantic large model of the transceiver to be trained by using the channel codec to be trained and the semantic codec based on the large model to be trained;

The training module 403 is used to cyclically and alternately train the codec and the projector related to the modality to be trained, and the channel codec to be trained, under the premise of keeping the parameters of the pre-trained large language model codec unchanged, to obtain a large semantic model of the transceiver;

The transmission and reconstruction module 404 is used to transmit and reconstruct the multimodal data using the semantic big model of the transmitting and receiving end.

The system specifically includes:

A first encoding module is used for the transmitter to jointly encode the multimodal data from the source through a semantic encoder based on a large model in the semantic large model of the transmitter and receiver, capture and fuse the semantic associations between different modalities, and generate a high-dimensional semantic space representation shared across modalities;

A second encoding module is used for the transmitter to convert the cross-modal shared high-dimensional semantic space representation into semantic symbols through an adaptive channel encoder in the semantic large model of the transmitter and receiver, select a coding rate matching the channel state for encoding, and select a transmission strategy matching the channel state for transmission;

A first decoding module is used for the receiving end to restore the received semantic symbols into a high-dimensional semantic space representation shared across modalities through an adaptive channel decoder in the semantic large model of the transceiver end, select a decoding rate matching the channel state for decoding, and select an error correction strategy matching the channel state for error correction;

The second decoding module is used for the receiving end to reconstruct the restored cross-modal shared high-dimensional semantic space representation through the semantic decoder based on the large model in the semantic large model of the transmitting and receiving end to obtain reconstructed multimodal data.

Wherein, the training module includes:

A first training submodule, used for freezing the parameters of the pre-trained large language model codec, and separately training the channel codec to be trained to adapt to the channel state;

A second training submodule is used to freeze the parameters of the channel codec trained to a certain degree, and train the parameters of the codec and projector related to the modality to be trained;

A cyclic alternating training submodule is used to cyclically alternate the first training submodule and the second training submodule to optimize the performance of the channel codec to be trained, the modality-related codec to be trained, and the modality-related projector until the loss function value of the semantic large model of the transceiver converges or meets the predetermined performance index.

Wherein, the training module also includes:

A difference calculation submodule, used for calculating the difference between the reconstructed communication intent of the reconstructed multimodal data and the actual communication intent of the multimodal data from the source during the training process;

An evaluation submodule, used for evaluating whether the semantic large model of the transmitting and receiving ends maintains semantic consistency according to the difference;

The adjustment submodule is used to adjust the parameters and training strategy of the semantic large model of the transmitting and receiving ends based on the evaluation results to reduce the difference.

Wherein, the system further comprises:

A preprocessing module, used to preprocess the multimodal data so that the preprocessed multimodal data has a preset data format;

The first encoding module is implemented by the large model-based semantic encoder, which includes a modal encoder, an input projector and a large language model encoder; the modal encoder is used to extract features from the preprocessed multimodal data; the input projector is used to align the extracted features so that data of different modalities can be effectively integrated in the same feature space; the large language model encoder is used to map data of different modalities into a unified semantic space to achieve semantic information fusion of multimodal data.

Among them, the large model-based semantic decoder in the second decoding module includes a modality decoder, an output projector and a large language model decoder, the large language model decoder is used to restore the high-dimensional semantic space representation from a unified semantic space into a first feature of different modal data; the output projector is used to decode the first feature of the different modal data to obtain a second feature of the different modal data; the modality decoder is used to reconstruct the second feature of the different modal data into the original multimodal data.

Wherein, the system further comprises:

An output result acquisition module is used to execute corresponding downstream tasks using a downstream task execution function component based on the multimodal data reconstructed by the large semantic model of the transmitting and receiving end to obtain an output result;

An optimization module, used for optimizing and adjusting the parameters of the downstream task execution functional component based on the output result to obtain an adjusted downstream task execution functional component;

The downstream task execution module is used to execute the corresponding task based on the user's interactive instructions and the reconstructed multimodal data through the adjusted downstream task execution functional component.

Based on the same application concept, the embodiment of the present application discloses an electronic device in the third aspect. Figure 5 shows a schematic diagram of an electronic device disclosed in the embodiment of the present application. As shown in Figure 5, the electronic device 100 includes: a memory 110 and a processor 120. The memory of the electronic device is not less than 12G, and the main frequency of the processor is not less than 2.4GHz. The memory 110 and the processor 120 are connected through bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a multimodal semantic communication method based on a large model disclosed in the embodiment of the present application.

Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program/instruction is stored. When the computer program/instruction is executed by a processor, a multimodal semantic communication method based on a large model disclosed in the embodiment of the present application is implemented.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

The embodiments of the present application are described with reference to the flowcharts and/or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

The above is a detailed introduction to a multimodal semantic communication method, system, device and medium based on a large model provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims (10)

1. A multimodal semantic communication method based on a large model, characterized in that the method comprises: Obtain a pre-trained large language model codec and multimodal data including text, image, audio, and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus; Using the modality-related codec and the modality-related projector to be trained, and the pre-trained large language model codec, construct a semantic codec based on the large model to be trained, and using the channel codec to be trained and the semantic codec based on the large model to be trained, construct a semantic large model of the transceiver to be trained; Under the premise of keeping the parameters of the pre-trained large language model codec unchanged, cyclically and alternately training the codec and projector related to the modality to be trained, and the channel codec to be trained, to obtain a large semantic model of the transceiver; The multimodal data is transmitted and reconstructed using the semantic big model of the transmitting and receiving ends. 2. The multimodal semantic communication method based on a large model according to claim 1, characterized in that the method specifically comprises: The transmitter jointly encodes the multimodal data from the source through the semantic encoder based on the large model in the semantic large model of the transmitter and receiver, captures and fuses the semantic associations between different modalities, and generates a high-dimensional semantic space representation shared across modalities; The transmitter converts the cross-modal shared high-dimensional semantic space representation into semantic symbols through an adaptive channel encoder in the semantic large model of the transmitter and receiver, selects a coding rate matching the channel state for encoding, and selects a transmission strategy matching the channel state for transmission; The receiving end restores the received semantic symbols into a high-dimensional semantic space representation shared across modalities through an adaptive channel decoder in the semantic large model of the transceiver end, selects a decoding rate matching the channel state for decoding, and selects an error correction strategy matching the channel state for error correction; The receiving end reconstructs the restored cross-modal shared high-dimensional semantic space representation through a semantic decoder based on the large model in the semantic large model of the transmitting and receiving end to obtain reconstructed multimodal data. 3. According to the large model-based multimodal semantic communication method of claim 1, it is characterized in that the training process of the semantic large model of the transmitting and receiving end is as follows: The first stage of training: freezing the parameters of the pre-trained large language model codec, and separately training the channel codec to be trained to adapt to the channel state; The second stage of training: freezing the parameters of the channel codec trained to a certain degree, and training the parameters of the codec and projector related to the modality to be trained; The first stage training and the second stage training are cyclically alternated to optimize the performance of the channel codec to be trained, the modality-related codec to be trained, and the modality-related projector, until the loss function value of the semantic large model of the transceiver converges or meets the predetermined performance index. 4. The method for multimodal semantic communication based on a large model according to claim 3, wherein the training process of the semantic large model of the transmitting and receiving end further comprises: During the training process, a difference between a reconstructed communication intent of the reconstructed multimodal data and an actual communication intent of the multimodal data from the source is calculated; According to the difference, evaluating whether the semantic big model of the sending and receiving end maintains semantic consistency; Based on the evaluation results, the parameters and training strategy of the semantic large model of the transmitting and receiving ends are adjusted to reduce the difference. 5. The method for multimodal semantic communication based on a large model according to claim 2, characterized in that before jointly encoding the multimodal data from the information source, the method comprises: Preprocessing the multimodal data so that the preprocessed multimodal data has a preset data format; The joint encoding of the multimodal data from the information source is implemented by the semantic encoder based on the large model, and the semantic encoder based on the large model includes a modality encoder, an input projector and a large language model encoder; The modal encoder is used to extract features from the preprocessed multimodal data; The input projector is used to align the extracted features so that data from different modalities can be effectively integrated in the same feature space; The large language model encoder is used to map data of different modalities into a unified semantic space to achieve semantic information fusion of multimodal data. 6. The multimodal semantic communication method based on a large model according to claim 2, characterized in that the semantic decoder based on the large model comprises a modality decoder, an output projector and a large language model decoder, and the semantic decoder based on the large model reconstructs the restored high-dimensional semantic space representation shared across modalities to obtain reconstructed multimodal data, including: The large language model decoder is used to restore the high-dimensional semantic space representation from a unified semantic space into a first feature of different modal data; The output projector is used to decode the first feature of the different modal data to obtain the second feature of the different modal data; The modality decoder is used to reconstruct the second features of the different modality data into the original multi-modality data. 7. The method for multimodal semantic communication based on a large model according to claim 1, characterized in that after the multimodal data is transmitted and reconstructed using the semantic large model of the transmitting and receiving end, the method comprises: Based on the multimodal data reconstructed by the large semantic model of the transmitting and receiving end, a downstream task execution function component is used to execute corresponding downstream tasks to obtain output results; Optimizing and adjusting the parameters of the downstream task execution functional component based on the output result to obtain an adjusted downstream task execution functional component; The adjusted downstream task execution functional component executes the corresponding task based on the user's interactive instructions and the reconstructed multimodal data. 8. A multimodal semantic communication system based on a large model, characterized in that the system comprises: An acquisition module is used to acquire a pre-trained large language model codec and multimodal data including text, image, audio and video, wherein the pre-trained large language model codec is obtained by pre-training with a corpus; A construction module is used to construct a semantic codec based on a large model to be trained by using the codec and projector related to the modality to be trained, and the pre-trained large language model codec, and to construct a semantic large model of the transceiver to be trained by using the channel codec to be trained and the semantic codec based on the large model to be trained; A training module is used to cyclically and alternately train the codec and projector related to the modality to be trained, and the channel codec to be trained, under the premise of keeping the parameters of the pre-trained large language model codec unchanged, to obtain a large semantic model of the transceiver; The transmission and reconstruction module is used to utilize the large semantic model of the transmitting and receiving end to transmit and reconstruct the multimodal data. 9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the large model-based multimodal semantic communication method described in any one of claims 1 to 7. 10. A computer-readable storage medium, characterized in that a computer program/instruction is stored thereon, and when the computer program/instruction is executed by a processor, the multimodal semantic communication method based on a large model described in any one of claims 1 to 7 is implemented.