WO2024125391A2

WO2024125391A2 - Ai model monitoring method and apparatus, ai model performance measurement method and apparatus, and device

Info

Publication number: WO2024125391A2
Application number: PCT/CN2023/137311
Authority: WO
Inventors: 施源; 孙鹏; 吴昊
Original assignee: 维沃移动通信有限公司
Priority date: 2022-12-15
Filing date: 2023-12-08
Publication date: 2024-06-20
Also published as: CN118214667A; WO2024125391A3

Abstract

Disclosed in the present application are an AI model monitoring method and apparatus, an AI model performance measurement method and apparatus, and a device. The AI model monitoring method comprises: a terminal acquiring monitoring reference information, and a measurement result of AI model monitoring for a network-side device, and determining label data according to the measurement result; and the terminal determining a monitoring result of an AI model according to the monitoring reference information and the label data, wherein the monitoring result is used for determining the performance of the AI model.

Description

AI model monitoring method, AI model performance measurement method, device and equipment

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese Patent Application No. 202211617140.4 filed in China on December 15, 2022, the entire contents of which are incorporated herein by reference.

Technical Field

The present application belongs to the field of communication technology, and specifically relates to an AI model monitoring method, an AI model performance measurement method, a device and equipment.

Background technique

When the Artificial Intelligence (AI) model is inferred on the network side, in order to realize the AI model monitoring function, the terminal (for example, User Equipment (UE)) not only needs to measure the label data, but also needs to feedback the label data and/or the input data for the AI model. The network side performs AI model inference based on the received AI model input data to obtain the inference result, and compares the inference result with the label data fed back by the UE to determine or update the performance of the AI model. This means that in order to monitor the performance of the AI model on the network side, the UE needs to feedback a large amount of label data and/or input data for the AI model, which greatly increases the feedback overhead of the UE.

Summary of the invention

The embodiments of the present application provide an AI model monitoring method, an AI model performance measurement method, an apparatus and equipment to solve the problem of high feedback overhead in the AI model monitoring process in related technologies.

In a first aspect, an AI model monitoring method is provided, comprising:

The terminal obtains monitoring reference information and measurement results of the AI model monitoring of the network side device, and determines the label data according to the measurement results;

The terminal determines a monitoring result of the AI model according to the monitoring reference information and the label data;

Wherein, the monitoring results are used to determine the performance of the AI model.

Second, a method for measuring the performance of an AI model is provided, including:

The network-side device sends monitoring reference information of the AI model;

The network side device receives the monitoring result of the AI model from the terminal;

The network side device determines the performance of the AI model according to the monitoring result;

Among them, the monitoring results of the AI model are obtained based on the monitoring reference information.

In a third aspect, an AI model monitoring device is provided, comprising:

A first acquisition module is used to acquire monitoring reference information and measurement results of AI model monitoring of network-side devices, and determine label data according to the measurement results;

The first determination module is used to determine the monitoring of the AI model according to the monitoring reference information and the label data. Test results.

In a fourth aspect, a device for measuring AI model performance is provided, comprising:

A fourth sending module, used to send monitoring reference information of the AI model;

A second receiving module, used to receive monitoring results of the AI model from a terminal;

A second determination module, used to determine the performance of the AI model according to the monitoring results;

In a fifth aspect, a communication device is provided, comprising: a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect or the second aspect.

In the sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by the processor of the terminal, the steps of the method described in the first aspect are implemented, or when the program or instruction is executed by the processor of the network side device, the steps of the method described in the second aspect are implemented.

In the seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method described in the first aspect or the second aspect.

In an eighth aspect, a computer program/program product is provided, wherein the computer program/program product is stored in a non-volatile storage medium, and the program/program product is executed by at least one processor to implement the steps of the method described in the first aspect or the second aspect.

In the ninth aspect, a communication system is provided, which includes a terminal and a network side device, wherein the terminal is used to execute the steps of the method described in the first aspect or the second aspect, and the network side device is used to execute the steps of the method described in the first aspect or the second aspect.

In an embodiment of the present application, the terminal obtains monitoring reference information and measurement results of the AI model monitoring of the network side device, and determines the label data based on the measurement results; the terminal determines the monitoring results of the AI model based on the monitoring reference information and the label data; wherein the monitoring results are used to determine the performance of the AI model, so that the terminal does not need to feedback the label data to the network side device and/or use it as input data for the AI model, thereby saving signaling overhead while ensuring the accuracy of AI model monitoring.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of a neural network;

Figure 2 is a schematic diagram of a neuron;

FIG3 is one of the schematic diagrams of beam prediction based on the AI model;

FIG4 is a second schematic diagram of beam prediction based on an AI model;

FIG5 is a third schematic diagram of beam prediction based on an AI model;

FIG6 is a schematic diagram of the architecture of a wireless communication system according to an embodiment of the present application;

FIG7 is a flow chart of an AI model monitoring method provided in an embodiment of the present application;

FIG8 is a flow chart of a method for measuring AI model performance according to an embodiment of the present application;

FIG9 is a schematic diagram of an AI model monitoring device provided in an embodiment of the present application;

FIG10 is a schematic diagram of a device for measuring AI model performance according to an embodiment of the present application;

FIG11 is a schematic diagram of a terminal provided in an embodiment of the present application;

FIG12 is a schematic diagram of a network side device provided in an embodiment of the present application;

FIG. 13 is a schematic diagram of a communication device provided in an embodiment of the present application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, but not to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship. The term "indication" in the specification and claims of the present application can be either an explicit indication or an implicit indication. Among them, an explicit indication can be understood as the sender explicitly notifying the receiver of the operation or request result to be performed in the indication sent; an implicit indication can be understood as the receiver making a judgment based on the indication sent by the sender and determining the operation or request result to be performed based on the judgment result.

It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE)/LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) and other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the above-mentioned systems and radio technologies as well as other systems and radio technologies. The following description describes a new radio (NR) system for example purposes, and NR terms are used in most of the following descriptions, but these technologies can also be applied to applications other than NR system applications, such as the ^6th Generation (6G) communication system.

In order to facilitate understanding of the implementation of the present application, the following technical points are first introduced below.

1. Introduction to neural networks.

Artificial intelligence has been widely used in various fields. There are many ways to implement AI modules, such as neural networks, decision trees, support vector machines, Bayesian classifiers, etc.

This application uses a neural network as an example for illustration, but does not limit the specific type of AI module. The structure of the neural network is shown in FIG1 .

The neural network is composed of neurons, and a schematic diagram of neurons is shown in Figure 2. Where a ₁ , a ₂ , … a _K are inputs, w is the weight (multiplicative coefficient), b is the bias (additive coefficient), σ(.) is the activation function, z = a ₁ w ₁ + … + a _k w _k + … + a _K w _K + b. Common activation functions include Sigmoid function, tanh function, Rectified Linear Unit (ReLU), etc.

The parameters of a neural network can be optimized using an optimization algorithm. An optimization algorithm is a type of algorithm that can minimize or maximize an objective function (sometimes called a loss function). The objective function is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, a neural network model f(.) is constructed. With the model, the predicted output f(x) can be obtained based on the input x, and the difference between the predicted value and the true value (f(x)-Y) can be calculated. This is the loss function. If the appropriate W, b is found to minimize the value of the above loss function, the smaller the loss value, the closer the model is to the actual situation.

At present, the common optimization algorithms are basically based on the error back propagation (BP) algorithm. The basic idea of the BP algorithm is that the learning process consists of two processes: the forward propagation of the signal and the back propagation of the error. During the forward propagation, the input sample is transmitted from the input layer, processed by each hidden layer layer by layer, and then transmitted to the output layer. If the actual output of the output layer does not match the expected output, it will enter the error back propagation stage. Error back propagation is to propagate the output error layer by layer through the hidden layer to the input layer in some form, and distribute the error to all units in each layer, so as to obtain the error signal of each layer unit, and this error signal is used as the basis for correcting the weights of each unit. This process of adjusting the weights of each layer of the signal forward propagation and error back propagation is repeated. The process of continuous adjustment of weights is the learning and training process of the network. This process continues until the error of the network output is reduced to an acceptable level, or until the pre-set number of learning times is reached.

Common optimization algorithms include Gradient Descent, Stochastic Gradient Descent (SGD), mini-batch gradient descent, Momentum, Stochastic Gradient Descent with momentum, Adaptive Gradient Descent (Adagrad), Adadelta (Adadelta is an improvement on the Adagrad algorithm), root mean square prop (RMSprop), Adaptive Momentum Estimation (Adam), etc.

When these optimization algorithms backpropagate errors, they all calculate the derivative or partial derivative of the current neuron based on the error or loss obtained from the loss function, add the influence of the learning rate, previous gradient, derivative or partial derivative, obtain the gradient, and pass the gradient to the previous layer.

2. About beam measurement and beam reporting.

Analog beamforming is full-bandwidth transmission, and each polarization direction array element on the panel of each high-frequency antenna array can only send analog beams in a time-division multiplexing manner. The shaping weight of the analog beam is achieved by adjusting the parameters of the RF front-end phase shifter and other devices.

Currently, in academia and industry, polling is usually used to train analog beamforming vectors, that is, each element of each polarization direction of each antenna panel sends training signals (i.e. candidate beamforming vectors) in turn at the agreed time in a time division multiplexing manner. After the measurement, the terminal feeds back a beam report, which is used by the network to implement simulated beam transmission in the next service transmission using the training signal. The content of the beam report usually includes the identifiers of the optimal transmission beams and the measured receiving power of each transmission beam.

The number of beam reports is determined by the parameters configured by the network to the terminal. The Radio Resource Control (RRC) configuration parameters are used to configure the number of reference signals (RS) and reference signal received power (RSRP) that should be included in the terminal's beam report. The values of the quantity configuration are 1, 2, 3, and 4, and the default value is 1. In addition, the quantity limit is based on the terminal's capabilities, and the terminal will first report the maximum number it can support.

When the terminal beam report contains only one layer 1 reference signal received power (L1-RSRP), a 7-bit quantization method is used, the quantization step is 1dB, and the quantization range is -140dBm to -44dBm. When the terminal is instructed that the beam report contains multiple L1-RSRPs, or group-based beam report is enabled, the strongest RSRP is quantized using 7-bit quantization, and the remaining RSRPs are quantized using 4-bit differential quantization, with a quantization step of 2dB.

3. About the use of AI methods for beam prediction.

One possible approach is shown in Figure 3. Using the RSRP of some beam pairs as input, the output of the AI model is the RSRP result of all beam pairs. A beam pair consists of a transmit beam and a receive beam. The number of inputs to the AI model is the number of selected beam pairs, and the number of outputs is the number of all beam pairs.

There is also an additional method to enhance the beam prediction performance as shown in FIG4 .

The associated information is added on the input side. The associated information is generally the angle-related information corresponding to the selected beam pairs for input, beam identification (ID) information, etc. Therefore, the number of inputs of this model is still related to the number of selected beam pairs, and the number of outputs is still equal to the number of all beam pairs.

There is also an improved method based on the above as shown in FIG5 .

It mainly affects the output of the AI model by changing the expected information through the AI model.

The input type of the AI model includes at least one of the following:

(1) Information related to beam quality;

(2) beam information;

(3) End A sends beam information;

(4) End B receives beam information;

(5) Beam information expected by the B end;

(6) The B-side receiving beam information expected by the B-side;

(7) The beam information that the B end expects the A end to send;

(8) Time-related information related to beam quality;

(9) Information related to the expected prediction time.

The beam quality information herein includes but is not limited to at least one of the following types: Layer 1 signal-to-noise and interference ratio (L1-SINR), L1-RSRP, Layer 1 reference signal received Quality (Reference Signal Received Quality, L1-RSRQ), Layer 3 signal-to-noise and interference ratio (Layer 3signal-to-noise and interference ratio, L3-SINR), Layer 3 reference signal received power (Layer 3reference signal received power, L3-RSRP), Layer 3 reference signal received quality (Reference Signal Received Quality, L3-RSRQ), etc.;

The beam information in this article refers to the associated information corresponding to the beam quality information contained in the beam report. The associated information includes but is not limited to at least one of the following: beam ID information, beam angle information, beam gain information, beam width information, expected information, etc.

Among them, the beam ID information is used to characterize the relevant information of the identity identification of the beam, including but not limited to at least one of the following: transmitting beam ID, receiving beam ID, beam ID, reference signal set (set) ID corresponding to the beam, reference signal resource (resource) ID corresponding to the beam, uniquely identified random ID, coding value after additional AI network processing, beam angle information, resource index information, channel state information reference signal resource indicator (CSI-RS Resource Indicator, CRI), synchronization signal block resource indication (SS/PBCH Block Resource Indicator, SSBRI), etc.

The beam angle information is used to characterize the angle information corresponding to the beam, including but not limited to at least one of the following: angle-related information, sending angle-related information, and receiving angle-related information.

The angle information is related information used to characterize the angle or identity, for example, angle, radian, index encoding value, ID value, encoding value after additional AI network processing, etc.

4. Regarding beam reporting and beam resource configuration.

The association relationships are as follows: beam report configuration is associated with resource configuration, resource configuration is associated with beam resource set configuration, and beam resource set configuration is associated with beam resource configuration.

For example, CSI report configuration (CSI-ReportConfig) is associated with CSI resource configuration (CSI-ResourceConfig), and CSI-ResourceConfig is associated with resource set (Resource Set) and time domain behavior.

Among them, (1) if the CSI-RS resource set is used, the corresponding one is the non-zero power (Non-Zero Power, NZP)-CSI-RS-Resource Set, in which the NZP-CSI-RS-Resource is associated, and the time domain behavior is used to indicate the time domain periodic attribute associated with the CSI-RS resource set.

(2) If the SSB resource set is used, the corresponding one is CSI-SSB-Resource Set, and the SSB index (Index) is associated in the Resource Set. At this time, the time domain behavior is invalid.

A CSI-ReportConfig (e.g., beam report configuration) contains up to three CSI-ResoureConfig (e.g., beam resource configuration), and the specific relationship is as follows:

(1) Aperiodic CSI-ReportConfig can be associated with periodic, semi-persistent, and semi-persistent CSI-ResourceConfig, and up to three beam resource configurations can be configured.

(a) When 1 CSI-ResourceConfig is configured, it is used for channel measurement (CM), for example, including L1-RSRP measurement.

(b) Two CSI-ResourceConfigs are configured, the first one is used for CM, and the second one is used for interference measurement (Interference Measurement, IM), for example, the second one is used for interference measurement of zero power resources.

(c) Three CSI-ResourceConfigs are configured, the first one is used for CM, the second one is used for IM, for example, the second one is used for interference measurement of zero power resources, and the third one is used for interference measurement, for example, the third one is used for interference measurement of non-zero power resources.

(2) Semi-persistent CSI-ReportConifg can be associated with periodic, semi-persistent CSI-ResourceConfig, and can configure up to 2 beam resource configurations.

(a) 1 CSI-ResourceConfig, used for CM channel measurement, such as L1-RSRP measurement.

(b) 2 CSI-ResourceConfigs, the first one is for CM and the second one is for IM, for example, the second one is used for interference measurement of zero power resources.

(3) Periodic CSI-ReportConfig can be associated with periodic and semi-continuous CSI-ResourceConfig, and can configure up to 2 beam resource configurations

Among them, the time domain behaviors of one or more CSI-ResourceConfigs associated with CSI-ReportConfig are consistent.

For periodic and semi-continuous CSI resourceConfig only one Resource set is supported. However, if group-based beam reporting is supported in the report, two sets can be configured.

For non-periodic CSI resourceConfig, there is no limit of 1 set and up to 16 sets can be configured.

A maximum of 64 NZP CSI-RS reousrces are supported in one CSI-RS resource set. When reportQuantity = 'none', 'cri-RI-CQI', 'cri-RSRP' or 'ssb-Index-RSRP', a maximum of 128 resources are supported in all CSI-RS resource sets.

The repetition information associated with the CSI-RS resource set, if configured to be on, the UE will assume that all CSI-RS resources in the CSI-RS resource set use the same transmit beam information when they are sent. If configured to be off, the UE will not assume that these resources use the same transmit beam information. That is, the repetition parameter in the CSI-RS resource set will control the beam information attributes of all resources associated with the resource set.

5. About model monitoring.

If the AI model is inferred on the network side, the terminal can feed back the measurement results to the network side. The feedback content includes at least the input of the model and the label data used for model monitoring, thereby realizing the model monitoring function on the network side.

FIG6 shows a block diagram of a wireless communication system applicable to an embodiment of the present application. The wireless communication system includes a terminal 61 and a network side device 62. The wireless communication system may be a communication system with wireless AI functions such as 5G-Advanced or 6G.

The terminal 61 may be a mobile phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR)/virtual reality (VR) device, a robot, a wearable device, a vehicle user device, or a wearable device. Equipment, VUE), pedestrian terminal (Pedestrian User Equipment, PUE), smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), game consoles, personal computers (personal computers, PCs), ATMs or self-service machines and other terminal-side devices, wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. In addition to the above-mentioned terminal devices, the terminal involved in the present application may also be a chip in the terminal, such as a modem chip or a system-on-chip (SoC). It should be noted that the specific type of terminal 61 is not limited in the embodiments of the present application.

The network side device 62 may include an access network device or a core network device, wherein the access network device may also be referred to as a wireless access network device, a wireless access network (Radio Access Network, RAN), a wireless access network function or a wireless access network unit. The access network device may include a base station, a wireless local area network (Wireless Local Area Networks, WLAN) access point or a wireless fidelity (Wireless Fidelity, WiFi) node, etc. The base station may be referred to as a node B, an evolved node B (evolved Node B, eNB), an access point, a base transceiver station (Base Transceiver Station, BTS), a radio base station, a radio transceiver, a basic service set (Basic Service Set, BSS), an extended service set (Extended Service Set, ESS), a home B node, a home evolved B node, a transmitting and receiving point (Transmitting Receiving Point, TRP) or other appropriate terms in the field, as long as the same technical effect is achieved, the base station is not limited to a specific technical vocabulary, it should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited. The core network equipment may include but is not limited to at least one of the following: core network nodes, core network functions, mobility management entity (Mobility Management Entity, MME), access and mobility management function (Access and Mobility Management Function, AMF), session management function (Session Management Function, SMF), user plane function (User Plane Function, UPF), policy control function (Policy Control Function, PCF), policy and charging rules function unit (Policy and Charging Rules Function, PCRF), edge application service discovery function (Edge Application Server Discovery ... user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), user plane function (User Plane Function, UPF), tion, EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (L-NEF), Binding Support Function (BSF), Application Function (AF), etc. It should be noted that in the embodiments of the present application, only the core network device in the NR system is taken as an example for introduction, and the specific type of the core network device is not limited.

The following, in combination with the accompanying drawings, describes in detail the AI model monitoring method, device, communication equipment and readable storage medium provided in the embodiments of the present application through some embodiments and their application scenarios.

Referring to Figure 7, an embodiment of the present application provides an AI model monitoring method, which is applied to a terminal, and the specific steps include: step 701 and step 702.

Step 701: The terminal obtains monitoring reference information and measurement results of the AI model monitoring of the network side device, and determines label data according to the measurement results;

The monitoring reference information in this article may also be referred to as monitoring reference values.

Optionally, the terminal may first obtain monitoring reference information of the AI model, and then obtain corresponding measurement results of monitoring the AI model of the network side device, and determine the label data based on the measurement results; or the terminal may first obtain measurement results of monitoring the AI model of the network side device, and determine the label data based on the measurement results, and then obtain the corresponding monitoring reference information of the AI model; or the terminal may simultaneously obtain the monitoring reference information and obtain corresponding measurement results of monitoring the AI model of the network side device, and determine the label data based on the measurement results.

Step 702: The terminal determines the monitoring result of the AI model according to the monitoring reference information and the label data;

Optionally, in this embodiment, the terminal may send the monitoring result to the network side device.

Optionally, the performance of the AI model includes but is not limited to: at least one of: the accuracy of the model's prediction of optimal beam information, the accuracy of the model's prediction of the top K (top-k) optimal beam information, the accuracy of the model's prediction of optimal beam quality information, the accuracy of the model's prediction of the top-k optimal beam information, whether the requirements are met, etc., where k is a positive integer.

In one implementation of the present application, the terminal obtains monitoring reference information, including:

The terminal receives the monitoring reference information sent by the network side device.

Optionally, the network side device may send monitoring reference information of the AI model periodically, non-periodically, or semi-continuously.

Optionally, the monitoring reference information of the AI model can be carried in signaling/information such as RRC, Medium Access Control (MAC) control element (CE), and downlink control information (DCI).

It should be noted that, in this embodiment, the manner in which the terminal receives monitoring reference information of the AI model from the network side device is not limited.

In one embodiment of the present application, the monitoring reference information includes: one or more reference information determined by the network side device based on the inference result of the AI model, which can improve the accuracy of AI model monitoring.

For example, the reference information includes but is not limited to at least one of the optimal beam index, the beam quality information of the optimal beam, the optimal beam information, the top-k optimal beam indexes, the beam quality information of the top-k optimal beams, the top-k optimal beam information, and the like.

In one embodiment of the present application, the method further includes:

The terminal receives first information, where the first information is used to indicate a time association relationship between the monitoring reference information and a measurement result.

In one embodiment of the present application, when the monitoring reference information is sent periodically and/or semi-continuously by the network side device,

The period and/or period offset of the monitoring reference information is related to the period and/or period offset of the beam measurement resources used for AI model monitoring.

In one embodiment of the present application, the period of the monitoring reference information satisfies at least one of the following: the period of the monitoring reference information is equal to the period used for the beam measurement resource, and the period of the monitoring reference information is less than the period of the beam measurement resource. The period of the bundle measurement resource;

and / or,

The periodic offset of the monitoring reference information is smaller than the periodic offset of the beam measurement resource.

In one implementation of the present application, when the monitoring reference information is sent aperiodically by a network side device, the method further includes:

The terminal expects the network side device to perform a first operation;

The first operation includes one of the following:

(1) When the network side device sends the monitoring reference information, the network side device triggers the aperiodic beam measurement resource for AI model monitoring, or when the network side device triggers the aperiodic beam measurement resource for AI model monitoring, the network side device aperiodically sends the monitoring reference information;

(2) the network side device aperiodically sends monitoring reference information, and the network side device triggers aperiodic beam measurement resources for AI model monitoring within a first time window, where the first time window is a time window after sending the monitoring reference information;

(3) the network side device sends monitoring reference information aperiodically, and the network side device sends aperiodic beam measurement resources for AI model monitoring within a second time window, where the second time window is a time window after sending the monitoring reference information;

(3) the network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information aperiodically within a third time window, where the third time window is a time window after the aperiodic beam measurement resources for AI model monitoring are triggered;

(4) The network side device triggers the sending of non-periodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information non-periodically within a fourth time window, and the fourth time window is a time window after sending the non-periodic beam measurement resources for AI model monitoring.

It can be understood that the first time window, the second time window, the third time window, and the fourth time window can be determined by at least one of the following methods: network configuration, protocol agreement, and terminal reporting.

In one implementation of the present application, the terminal determines the monitoring result of the AI model according to the monitoring reference information and the label data, including:

The terminal determines the monitoring result of the AI model according to the label data and the most recently obtained monitoring reference information;

or,

The terminal determines the monitoring result of the AI model based on the label data and the monitoring reference information most recently obtained within the fifth time window.

It can be understood that the fifth time window can be determined by at least one of the following methods: configuration on the network side, agreement on the protocol, reporting by the terminal, etc.

In one implementation of the present application, the reference time of the first time window, the second time window, the third time window, the fourth time window or the fifth time window includes one of the following:

(1) the time when the terminal performs corresponding beam measurement;

(2) The terminal determines the time of tag data according to the measurement result;

(3) the time when the terminal receives the monitoring reference information;

(4) The time at which the terminal receives the effective time of the monitoring reference information.

In one embodiment of the present application, the method further includes:

The terminal sends a first measurement result to the network side device, where the first measurement result includes at least part of the measurement results monitored by the AI model;

Among them, the AI model of the network side device is used to determine the monitoring reference information according to the first measurement result.

For example, the terminal performs beam measurement to determine a first measurement result and label data, and the terminal feeds back the first measurement result to the network side device. After receiving the first measurement result, the network side device performs AI model inference, and determines monitoring reference information based on the inference result. The network side device sends the corresponding monitoring reference information to the terminal, and the terminal determines the monitoring result of the AI model based on the label data and the corresponding monitoring reference information. The terminal sends the monitoring result of the AI model to the network side device, and the network side device determines the performance of the AI model based on the monitoring result of the AI model.

In one implementation manner of the present application, the beam information of the first measurement result is the same as at least one of the following:

(1) beam information of the measurement results contained in a beam measurement feedback report, where the beam measurement feedback report is related to AI model monitoring;

(2) Beam information indicated by the network side device.

In one embodiment of the present application, the monitoring reference information is associated with the most recently obtained measurement result of the AI model monitoring.

In one implementation of the present application, the terminal receives the monitoring reference information sent by the network side device, including:

The terminal receives the monitoring reference information sent by the network side device within a sixth time window;

The sixth time window is a time window after the terminal performs beam measurement or the terminal sends the first measurement result.

In one embodiment of the present application, the method further includes:

The terminal sends second information, where the second information is used to indicate at least one of the following: the network side device resends the monitoring reference information, re-triggers the beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.

For example, if the terminal does not determine the monitoring reference information of the corresponding AI model within the fifth time window or the sixth time window, the terminal sends the second information.

In one embodiment of the present application, the method further includes:

The terminal caches the first measurement result within a seventh time window, where the seventh time window is a time window after the terminal performs beam measurement to obtain the first measurement result or the terminal sends the first measurement result.

In one embodiment of the present application, the method further includes:

The terminal caches the tag data within an eighth time window, where the eighth time window is a time window after the terminal performs beam measurement to obtain the first measurement result or determines the tag data or sends the first measurement result.

It can be understood that the sixth time window, the seventh time window, and the eighth time window can be determined by at least one of the following methods: configuration on the network side, agreement on the protocol, and reporting by the terminal.

Optionally, the use of the above time window may be one of the starting position, the ending position, and the special position of the time unit, wherein the time unit may be a unit representing time such as a time slot or a symbol.

Optionally, the first, second, third, fourth, fifth, sixth, seventh, and eighth time windows may be of the same time window length, or of different time window lengths.

In one implementation of the present application, the terminal sends the monitoring result to the network side device, including:

The terminal sends a monitoring feedback report of the AI model to a network side device, where the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring result is included in the monitoring feedback report.

In one implementation of the present application, the third information is used to indicate that the monitoring feedback report does not include the monitoring result, and the monitoring feedback report includes the label data.

Referring to Figure 8, an embodiment of the present application provides a method for measuring AI model performance, which is applied to a network side device, and the specific steps include: step 801, step 802 and step 803.

Step 801: The network side device sends monitoring reference information of the AI model;

Step 802: The network-side device receives monitoring results of the AI model from the terminal;

Step 803: The network-side device determines the performance of the AI model according to the monitoring result;

In one embodiment of the present application, the monitoring reference information includes: one or more reference information determined by the network side device according to the inference result of the AI model.

In one implementation of the present application, the reference information includes at least one of the following:

Optimal beam index;

Beam quality information of the optimal beam;

Optimal beam information;

Top k best beam indices;

Beam quality information of the first k best beams;

The first k best beam information;

Wherein, k is a positive integer.

In one embodiment of the present application, the method further includes:

The network side device sends first information, where the first information is used to indicate a time correlation relationship between the monitoring reference information and a measurement result monitored by the AI model.

In one embodiment of the present application, the period of the monitoring reference information satisfies at least one of the following: the period of the monitoring reference information is equal to the period of the beam measurement resource, and the period of the monitoring reference information is less than the period of the beam measurement resource;

and / or,

In one implementation of the present application, the network side device sends monitoring reference information of the AI model, including:

The network side device aperiodically sends monitoring reference information, and the network side device triggers aperiodic beam measurement resources corresponding to the aperiodic sending of monitoring reference information;

Among them, the non-periodic beam measurement resources are used to obtain label data of the AI model.

In one embodiment of the present application, while the network side device aperiodically sends monitoring reference information, the network side device triggers aperiodic beam measurement resources for AI model monitoring, or, while the network side device triggers aperiodic beam measurement resources for AI model monitoring, the network side device aperiodically sends monitoring reference information.

In one embodiment of the present application, the trigger status indication information of the non-periodic beam measurement resource used for AI model monitoring is associated with the monitoring reference information sent non-periodically by the network side device.

In one implementation of the present application, the network side device aperiodically sends monitoring reference information, and the network side device triggers aperiodic beam measurement resources corresponding to the aperiodic sending of monitoring reference information, including:

The network side device sends monitoring reference information aperiodically;

The network side device triggers a non-periodic beam measurement resource for AI model monitoring within a first time window, where the first time window is a time window after sending the monitoring reference information;

or,

The network side device sends monitoring reference information aperiodically;

The network side device sends a non-periodic beam measurement resource for AI model monitoring within a second time window, where the second time window is a time window after sending the monitoring reference information;

or,

The network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring;

The network side device sends monitoring reference information aperiodically within a third time window, where the third time window is a time window after triggering the aperiodic beam measurement resource;

or,

The network side device sends monitoring reference information aperiodically within a fourth time window, and the fourth time window is a time window after sending the aperiodic beam measurement resource.

In one embodiment of the present application, the method further includes:

The network side device receives second information, where the second information is used to indicate at least one of the following: the network side device resends monitoring reference information, re-triggers beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.

In one embodiment of the present application, the method further includes:

The network side device receives a first measurement result sent by the terminal, where the first measurement result is at least a part of the measurement results monitored by the AI model;

The network side device determines the monitoring reference information according to the AI model and the first measurement result, where the first measurement result is a measurement result related to the AI model input.

In one embodiment of the present application, the network side device receives the monitoring result of the AI model including:

The network side device receives a monitoring feedback report of the AI model sent by the terminal, where the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring result is included in the monitoring feedback report.

In an embodiment of the present application, a network side device receives the monitoring results of the AI model sent by the terminal, and then determines the performance of the AI model based on the monitoring results. In this way, the terminal does not need to feedback label data and/or input data of the AI model to the network side device, thereby saving signaling overhead while ensuring the accuracy of AI model monitoring.

Referring to FIG. 9 , an embodiment of the present application provides an AI model monitoring device, which is applied to a terminal. The device 900 includes:

The first acquisition module 901 is used to acquire monitoring reference information and measurement results of the AI model monitoring of the network side device, and determine the label data according to the measurement results;

The first determination module 902 is used to determine the monitoring result of the AI model based on the monitoring reference information and the label data.

In one implementation of the present application, the device further includes: a first sending module, configured to send the monitoring result.

In an implementation manner of the present application, the first acquisition module 901 is further used to: receive the monitoring reference information sent by the network side device.

Optimal beam index;

Beam quality information of the optimal beam;

Optimal beam information;

Top k best beam indices;

Beam quality information of the first k best beams;

The first k best beam information;

Wherein, k is a positive integer.

In one embodiment of the present application, the monitoring reference information includes: one or more reference information determined by the network side device based on the inference result of the AI model.

In one embodiment of the present application, the device further includes:

The first receiving module is used to receive first information, where the first information is used to indicate the time correlation relationship between the monitoring reference information and the measurement results monitored by the AI model.

In one embodiment of the present application, the period of the monitoring reference information satisfies at least one of the following: the period of the monitoring reference information is equal to the period used for the beam measurement resource, and the period of the monitoring reference information is less than the period of the beam measurement resource;

and / or,

In one implementation of the present application, when the monitoring reference information is sent aperiodically by a network side device, the apparatus further includes:

A processing module, configured to expect the network-side device to perform a first operation;

The first operation includes one of the following:

(3) the network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information aperiodically within a third time window, where the third time window is a time window after the aperiodic beam measurement resources are triggered;

(4) The network side device triggers the sending of non-periodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information non-periodically within a fourth time window, and the fourth time window is a time window after sending the non-periodic beam measurement resources.

In one implementation of the present application, the first determining module 902 is further configured to:

Determine the monitoring result of the AI model based on the label data and the most recently obtained monitoring reference information;

or,

The monitoring result of the AI model is determined based on the label data and the most recently obtained monitoring reference information within the fifth time window.

(1) the time when the terminal performs corresponding beam measurement;

(3) the time when the terminal receives the monitoring reference information;

In one embodiment of the present application, the device further includes:

The second sending module is used to send second information, where the second information is used to indicate at least one of the following: the network side device resends the monitoring reference information, re-triggers the beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.

In one embodiment of the present application, the device further includes:

A third sending module, configured to send a first measurement result to the network side device, where the first measurement result includes at least part of the measurement results monitored by the AI model;

(1) beam information of the measurement results contained in the beam measurement feedback report, where the beam measurement feedback report is related to AI model monitoring;

(2) Beam information indicated by the network side device.

In one embodiment of the present application, the device further includes:

A cache module is used to cache the first measurement result within a seventh time window, where the seventh time window is a time window after the terminal performs beam measurement to obtain the first measurement result or after the terminal sends the first measurement result.

In one embodiment of the present application, the device further includes:

In one embodiment of the present application, the first sending module is further used to: send a monitoring feedback report of the AI model to a network side device, wherein the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring result is included in the monitoring feedback report.

The device provided in the embodiment of the present application can implement each process implemented by the method embodiment of Figure 7 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

Referring to FIG. 10 , an embodiment of the present application provides a device for measuring AI model performance, which is applied to a network side device. The device 1000 includes:

The fourth sending module 1001 is used to send monitoring reference information of the AI model;

A second receiving module 1002 is used to receive a monitoring result of the AI model from a terminal;

A second determination module 1003, used to determine the performance of the AI model according to the monitoring results;

Optimal beam index;

Beam quality information of the optimal beam;

Optimal beam information;

Top k best beam indices;

Beam quality information of the first k best beams;

The first k best beam information;

Wherein, k is a positive integer.

In one embodiment of the present application, the device further includes:

The fifth sending module is used to send first information, where the first information is used to indicate the time correlation relationship between the monitoring reference information and the measurement results monitored by the AI model.

The period and/or period offset of the monitoring reference information is related to the period and/or period offset of the beam measurement resource.

and / or,

In one embodiment of the present application, the fourth sending module 1001 is further used for: non-periodic sending of monitoring reference information, and the network side device triggers non-periodic beam measurement resources corresponding to the non-periodic sending of monitoring reference information; wherein the non-periodic beam measurement resources are used to obtain label data of the AI model.

In one implementation of the present application, the trigger status indication information of the non-periodic beam measurement resource is associated with the monitoring reference information sent aperiodically by the network side device.

In one implementation of the present application, the fourth sending module 1001 is further configured to:

Aperiodic sending of monitoring reference information;

Triggering a non-periodic beam measurement resource for AI model monitoring within a first time window, where the first time window is a time window after sending monitoring reference information;

or,

Aperiodic sending of monitoring reference information;

Sending a non-periodic beam measurement resource for AI model monitoring within a second time window, where the second time window is a time window after sending the monitoring reference information;

or,

Triggering the sending of aperiodic beam measurement resources for AI model monitoring;

aperiodically sending monitoring reference information within a third time window, wherein the third time window is a time window after triggering the aperiodic beam measurement resource;

or,

The monitoring reference information is sent aperiodically within a fourth time window, and the fourth time window is a time window after the aperiodic beam measurement resource is sent.

In one embodiment of the present application, the device further includes:

The third receiving module is used to receive second information, where the second information is used to indicate at least one of the following: the network side device resends the monitoring reference information, re-triggers the beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.

In one embodiment of the present application, the device further includes:

A fourth receiving module, configured to receive a first measurement result sent by a terminal, where the first measurement result is at least a portion of the measurement results monitored by the AI model;

A third determination module is used to determine the monitoring reference information according to the AI model and the first measurement result. The first measurement result is a measurement result related to the AI model input.

In one embodiment of the present application, the second receiving module is further used to: receive a monitoring feedback report of the AI model sent by the terminal, wherein the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring result is included in the monitoring feedback report.

The device provided in the embodiment of the present application can implement each process implemented by the method embodiment of Figure 8 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

Fig. 11 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application. The terminal 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and at least some of the components in the processor 1110.

Those skilled in the art will appreciate that the terminal 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, so as to implement functions such as charging, discharging, and power consumption management through the power management system. The terminal structure shown in FIG11 does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.

It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

In the embodiment of the present application, after receiving downlink data from the network side device, the RF unit 1101 can transmit the data to the processor 1110 for processing; in addition, the RF unit 1101 can send uplink data to the network side device. Generally, the RF unit 1101 includes but is not limited to an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

The memory 1109 can be used to store software programs or instructions and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both a volatile and a non-volatile memory, or the memory 1109 may include a non-volatile memory. Among them, the non-volatile memory or non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (Erasable PROM, The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

The terminal provided in the embodiment of the present application can implement each process implemented in the method embodiment of Figure 7 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

Please refer to FIG. 12, which is a structural diagram of a network side device used in an embodiment of the present application. As shown in FIG. 12, the network side device 1200 includes: a processor 1201, a transceiver 1202, a memory 1203 and a bus interface, wherein the processor 1201 may be responsible for managing the bus architecture and general processing. The memory 1203 may store data used by the processor 1201 when performing operations.

In the embodiment of the present application, the network side device 1200 also includes: a program stored in the memory 1203 and executable on the processor 1201, and when the program is executed by the processor 1201, the steps in the method shown in FIG. 8 above are implemented.

In FIG. 12 , the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1201 and memory represented by memory 1203. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1202 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.

As shown in Figure 13, an embodiment of the present application also provides a communication device 1300, including a processor 1301 and a memory 1302, and the memory 1302 stores programs or instructions that can be run on the processor 1301. For example, when the communication device 1300 is a terminal, the program or instruction is executed by the processor 1301 to implement the various steps of the method embodiment of Figure 7 above. When the communication device 1300 is a network side device, the program or instruction is executed by the processor 1301 to implement the various steps of the method embodiment of Figure 8 above and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the method of Figure 7 or Figure 8 and the various processes of the above-mentioned embodiments are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes Computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disk, etc.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes shown in Figure 7 or Figure 8 and the various method embodiments mentioned above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

The embodiments of the present application further provide a computer program/program product, which is stored in a storage medium, and is executed by at least one processor to implement the various processes shown in Figure 7 or Figure 8 and the various method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described here.

An embodiment of the present application also provides a communication system, which includes a terminal and a network side device. The terminal is used to execute the various processes as shown in Figure 7 and the various method embodiments described above, and the network side device is used to execute the various processes as shown in Figure 8 and the various method embodiments described above, and can achieve the same technical effect. In order to avoid repetition, it will not be repeated here.

It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

An AI model monitoring method, comprising:

The terminal obtains monitoring reference information and measurement results of the AI model monitoring of the network side device, and determines the label data according to the measurement results;

The terminal determines a monitoring result of the AI model according to the monitoring reference information and the label data;

Wherein, the monitoring results are used to determine the performance of the AI model.
The method according to claim 1, wherein the terminal obtains monitoring reference information, comprising:

The terminal receives the monitoring reference information sent by the network side device.
The method according to claim 2, wherein the monitoring reference information includes: one or more reference information determined by the network side device based on the inference result of the AI model.
The method according to claim 3, wherein the reference information includes at least one of the following:

Optimal beam index;

Beam quality information of the optimal beam;

Optimal beam information;

Top k best beam indices;

Beam quality information of the first k best beams;

The first k best beam information;

Wherein, k is a positive integer.
The method according to claim 1, wherein the method further comprises:

The terminal sends the monitoring result to the network side device.
The method according to claim 1, wherein the method further comprises:

The terminal receives first information, where the first information is used to indicate a time association relationship between the monitoring reference information and the measurement result.
The method according to claim 2, wherein, in the case where the monitoring reference information is sent periodically and/or semi-continuously by the network side device,

The period and/or period offset of the monitoring reference information is related to the period and/or period offset of the beam measurement resources used for AI model monitoring.
The method according to claim 7, wherein the period of the monitoring reference information satisfies at least one of the following: the period of the monitoring reference information is equal to the period used for the beam measurement resource, and the period of the monitoring reference information is less than the period of the beam measurement resource;

and / or,

The periodic offset of the monitoring reference information is smaller than the periodic offset of the beam measurement resource.
The method according to claim 1, wherein, when the monitoring reference information is sent aperiodically by the network side device, the method further comprises:

The terminal expects the network side device to perform a first operation;

The first operation includes:

While the network side device sends the monitoring reference information, the network side device triggers the aperiodic beam measurement resource for AI model monitoring, or while the network side device triggers the aperiodic beam measurement resource for AI model monitoring, the network side device aperiodically sends the monitoring reference information;

or,

The network side device aperiodically sends monitoring reference information, and the network side device triggers aperiodic beam measurement resources for AI model monitoring within a first time window, where the first time window is a time window after sending the monitoring reference information;

or,

The network side device aperiodically sends monitoring reference information, and the network side device sends aperiodic beam measurement resources for AI model monitoring within a second time window, where the second time window is a time window after sending the monitoring reference information;

or,

The network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information aperiodically within a third time window, where the third time window is a time window after the aperiodic beam measurement resources are triggered;

or,

The network side device triggers the sending of non-periodic beam measurement resources for AI model monitoring, and the network side device sends monitoring reference information non-periodically within a fourth time window, and the fourth time window is a time window after sending the non-periodic beam measurement resources.
The method according to claim 1, wherein the terminal determines the monitoring result of the AI model according to the monitoring reference information and the label data, comprising:

The terminal determines the monitoring result of the AI model according to the label data and the most recently obtained monitoring reference information;

or,

The terminal determines the monitoring result of the AI model based on the label data and the monitoring reference information most recently obtained within the fifth time window.
The method according to claim 9 or 10, wherein the reference time of the first time window, the second time window, the third time window, the fourth time window or the fifth time window comprises one of the following:

The time when the terminal performs corresponding beam measurement;

The terminal determines the time of tag data according to the measurement result;

The time when the terminal receives the monitoring reference information;

The terminal receives the effective time of the monitoring reference information.
The method according to claim 1, wherein the method further comprises:

The terminal sends second information, where the second information is used to indicate at least one of the following: the network side device resends the monitoring reference information, re-triggers the beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.
The method according to claim 1, wherein the method further comprises:

The terminal sends a first measurement result to the network side device, where the first measurement result includes at least part of the measurement results monitored by the AI model;

Among them, the AI model of the network side device is used to determine the monitoring reference information according to the first measurement result.
The method according to claim 13, wherein the beam information of the first measurement result is the same as at least one of the following:

Beam information of measurement results included in a beam measurement feedback report related to AI model monitoring;

Beam information indicated by the network side device.
The method according to claim 1 or 13, wherein the monitoring reference information is associated with the most recently obtained measurement results of the AI model monitoring.
The method according to claim 2 or 13, wherein the terminal receives the monitoring reference information sent by the network side device, comprising:

The terminal receives the monitoring reference information sent by the network side device within a sixth time window;

The sixth time window is a time window after the terminal performs beam measurement or the terminal sends the first measurement result.
The method according to claim 13, wherein the method further comprises:

The terminal caches the first measurement result within a seventh time window, where the seventh time window is a time window after the terminal performs beam measurement to obtain the first measurement result or the terminal sends the first measurement result.
The method according to claim 5 or 13, wherein the method further comprises:

The terminal caches the tag data within an eighth time window, where the eighth time window is a time window after the terminal performs beam measurement to obtain the first measurement result or determines the tag data or sends the first measurement result.
The method according to claim 5, wherein the terminal sends the monitoring result to the network side device, comprising:

The terminal sends a monitoring feedback report of the AI model to a network side device, where the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring result is included in the monitoring feedback report.
The method according to claim 19, wherein the third information is used to indicate that the monitoring result is not included in the monitoring feedback report, and the monitoring feedback report includes the label data.
A method for measuring AI model performance, comprising:

The network-side device sends monitoring reference information of the AI model;

The network side device receives the monitoring result of the AI model from the terminal;

The network side device determines the performance of the AI model according to the monitoring result;

Among them, the monitoring results of the AI model are obtained based on the monitoring reference information.
The method according to claim 21, wherein the monitoring reference information includes: one or more reference information determined by the network side device based on the inference result of the AI model.
The method according to claim 22, wherein the reference information includes at least one of the following:

Optimal beam index;

Beam quality information of the optimal beam;

Optimal beam information;

Top k best beam indices;

Beam quality information of the first k best beams;

The first k best beam information;

Wherein, k is a positive integer.
The method according to claim 21, wherein the method further comprises:

The network side device sends first information, where the first information is used to indicate a time correlation relationship between the monitoring reference information and a measurement result monitored by the AI model.
The method according to claim 21, wherein, in the case where the monitoring reference information is sent periodically and/or semi-continuously by the network side device,

The period and/or period offset of the monitoring reference information is related to the period and/or period offset of the beam measurement resources used for AI model monitoring.
The method according to claim 25, wherein the period of the monitoring reference information satisfies at least one of the following: the period of the monitoring reference information is equal to the period of the beam measurement resource, and the period of the monitoring reference information is less than the period of the beam measurement resource;

and / or,

The periodic offset of the monitoring reference information is smaller than the periodic offset of the beam measurement resource.
The method according to claim 21, wherein the network side device sends the monitoring reference information of the AI model, including:

The network side device aperiodically sends monitoring reference information, and the network side device triggers aperiodic beam measurement resources corresponding to the aperiodic sending of monitoring reference information;

Among them, the non-periodic beam measurement resources are used to obtain label data of the AI model.
The method according to claim 27, wherein, while the network side device aperiodically sends monitoring reference information, the network side device triggers aperiodic beam measurement resources for AI model monitoring, or, while the network side device triggers aperiodic beam measurement resources for AI model monitoring, the network side device aperiodically sends monitoring reference information.
The method according to claim 28, wherein the trigger status indication of the non-periodic beam measurement resource The information is associated with the monitoring reference information sent aperiodically by the network side device.
The method according to claim 27, wherein the network side device aperiodically sends monitoring reference information, and the network side device triggers the aperiodic beam measurement resource corresponding to the aperiodic sending of the monitoring reference information, comprising:

The network side device sends monitoring reference information aperiodically;

The network side device triggers a non-periodic beam measurement resource for AI model monitoring within a first time window, where the first time window is a time window after sending the monitoring reference information;

or,

The network side device sends monitoring reference information aperiodically;

The network side device sends a non-periodic beam measurement resource for AI model monitoring within a second time window, where the second time window is a time window after sending the monitoring reference information;

or,

The network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring;

The network side device sends monitoring reference information aperiodically within a third time window, where the third time window is a time window after triggering the aperiodic beam measurement resource;

or,

The network side device triggers the sending of aperiodic beam measurement resources for AI model monitoring;

The network side device sends monitoring reference information aperiodically within a fourth time window, and the fourth time window is a time window after sending the aperiodic beam measurement resource.
The method according to claim 21, wherein the method further comprises:

The network side device receives second information, where the second information is used to indicate at least one of the following: the network side device resends monitoring reference information, re-triggers beam measurement resources for AI model monitoring, and the AI model monitoring of the network side device fails.
The method according to claim 21, wherein the method further comprises:

The network side device receives a first measurement result sent by the terminal, where the first measurement result is at least a part of the measurement results monitored by the AI model;

The network side device determines the monitoring reference information according to the AI model and the first measurement result, where the first measurement result is a measurement result related to the AI model input.
The method according to claim 21, wherein the network-side device receives the monitoring result of the AI model comprises:

The network side device receives a monitoring feedback report of the AI model sent by the terminal, where the monitoring feedback report includes third information, and the third information is used to indicate whether the monitoring feedback report includes a monitoring result of the AI model.
The method according to claim 33, wherein the third information is used to indicate that the monitoring feedback report does not include the monitoring results of the AI model, and the monitoring feedback report includes label data.
An AI model monitoring device, comprising:

A first acquisition module is used to acquire monitoring reference information and measurement results of AI model monitoring of network-side devices, and determine label data according to the measurement results;

The first determination module is used to determine the monitoring result of the AI model based on the monitoring reference information and the label data.
A device for measuring AI model performance, comprising:

A fourth sending module, used to send monitoring reference information of the AI model;

A second receiving module, used to receive monitoring results of the AI model from a terminal;

A second determination module, used to determine the performance of the AI model according to the monitoring results;

Among them, the monitoring results of the AI model are obtained based on the monitoring reference information.
A communication device comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method as claimed in any one of claims 1 to 34.
A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor of a terminal, implements the steps of a method as described in any one of claims 1 to 20, or, when executed by a processor of a network-side device, implements the steps of a method as described in any one of claims 21 to 34.