WO2023143264A1

WO2023143264A1 - Data compression method and apparatus

Info

Publication number: WO2023143264A1
Application number: PCT/CN2023/072668
Authority: WO
Inventors: 孙超; 程剑; 张亮; 夏晋伟; 廖有清
Original assignee: 华为技术有限公司
Priority date: 2022-01-30
Filing date: 2023-01-17
Publication date: 2023-08-03
Also published as: CN116567079A

Abstract

Disclosed are a data compression method and apparatus. The method comprises: a management device receives a message from a first network device, the message comprising a plurality of first data items, the plurality of first data items having a one-to-one correspondence with on a plurality of records, and the plurality of first data items being respectively from the corresponding records; the management device determines a first compression policy on the basis of the plurality of first data items; and the management device sends the first compression policy to the first network device. In this way, in the embodiments of the present application, the management device can determine a compression policy for the data items corresponding to the plurality of records, and can better consider the redundancy among the plurality of data items corresponding thereto, so that the compression rate can be high, compressed data can occupy less bandwidth, and costs can be saved.

Description

Data compression method and device

This application claims the priority of the Chinese patent application with the application number 202210113903.5 and the title of the invention "Data Compression Method and Device" filed with the China Patent Office on January 30, 2022, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the field of computer networks, and more specifically, to a data compression method and device.

Background technique

In a communication network, network devices such as distributed probes need to upload data to management devices such as cloud security centers. These data involve various aspects in the network scene, and the data volume is relatively large. Therefore, it is necessary to compress the data and transmit the compressed data, so as to reduce the bandwidth occupied by the interactive data.

Contents of the invention

The present application provides a data compression scheme, which uses the same compression strategy for multiple data items corresponding to multiple records, which can improve the compression rate and further reduce bandwidth occupation.

In a first aspect of the present application, a data compression method is provided. The method can be performed by a management device. The management device receives the message from the first network device. The message includes a plurality of first data items. The correspondence between the plurality of first data items and the plurality of records is a one-to-one correspondence, and the plurality of first data items come from corresponding records respectively. The management device determines a first compression strategy based on the plurality of first data items, and sends the first compression strategy to the first network device.

In this way, the management device can determine the compression strategy for the data items corresponding to multiple records, which can better reduce the redundancy between the corresponding data items, so that the compression rate is higher, which can make the compressed data occupy less space. Bandwidth saves costs.

In an implementation manner of the first aspect, the multiple first data items are compressed based on the second compression strategy, and the management device uses the second compression strategy to decompress the multiple first data items.

In this way, the management device can decompress the data items based on the compression policy, so that correct data items can be obtained.

In an implementation manner of the first aspect, the management device determines the second compression policy, and sends the second compression policy to the network device.

In an implementation manner of the first aspect, the management device receives the second compression strategy from the network device, and the second compression strategy is determined by the network device.

In an implementation manner of the first aspect, the management device determining the first compression policy based on the multiple first data items includes: when a condition is met, the management device determines the first compression policy based on the decompressed multiple first data items.

In an implementation manner of the first aspect, the condition includes: a time period elapsed after the management device determines the second compression policy reaches or exceeds a preset time period. In this way, the management device can periodically determine the compression policy, so as to update the compression policy.

In an implementation manner of the first aspect, the condition includes: the data types of the plurality of data items used to determine the second compression policy are different from the data types of the plurality of first data items, or the data types used to determine the second compression strategy The distribution characteristics of the plurality of data items of the compression strategy are different from the distribution characteristics of the plurality of first data items. In this way, the management device can update the compression strategy when the type or distribution characteristic of the data item changes, so that the used compression strategy can be adapted to the type or distribution characteristic of the data item, achieving a greater compression rate.

In an implementation manner of the first aspect, the sending the first compression strategy by the management device to the first network device includes: when the first compression strategy is different from the second compression strategy, the management device sends the first compression strategy to the first network device . In this manner, when it is determined that the compression strategy changes, the re-determined compression strategy is sent to the network device, which can save signaling overhead, reduce bandwidth occupation, and reduce costs.

In an implementation manner of the first aspect, the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second compression parameter.

In an implementation manner of the first aspect, the first compression algorithm or the second compression algorithm includes at least one of the following: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding and A compression method combining entropy coding, or a compression method combining difference coding and entropy coding.

In this way, the compression algorithm includes not only a single compression method, but also a combination of at least two different compression methods, which can achieve a greater compression ratio.

In an implementation manner of the first aspect, the management device sends the first compression strategy to the second network device, so that the second network device compresses multiple second data items based on the first compression strategy. The data type of the plurality of second data items is the same as the data type of the plurality of first data items.

In this way, the compression strategy determined based on one network device can also be used for other network devices, avoiding resource occupation caused by separately determining the compression strategy for each network device, and thus realizing effective utilization of resources at the management device. In this way, when the data of the second network device has not been collected or enough data of the second network device has not been collected at the management device, the management device may base on the data of other network devices having the same data type as the second network device. The data determines the compression strategy used by the second network device, so that the second network device can immediately compress the data based on the compression strategy to save bandwidth.

In a second aspect of the present application, a data compression method is provided. The method can be performed by a network device. The network device sends a message to the management device, and receives the first compression policy from the management device. The message includes multiple data items, and the corresponding relationship between the multiple data items and the multiple records is a one-to-one correspondence, and the multiple data items come from the corresponding records respectively. A first compression strategy is determined based on a plurality of data items.

In an implementation form of the second aspect, the network device compresses the data item using a second compression strategy, wherein the message includes the compressed data item.

In an implementation manner of the second aspect, the network device receives the second compression policy from the management device.

In an implementation manner of the second aspect, the network device determines the second compression policy, and sends the second compression policy to the management device.

In an implementation manner of the second aspect, the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second compression parameter.

In an implementation manner of the second aspect, the first compression algorithm or the second compression algorithm includes at least one of the following: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding and A compression method combining entropy coding, or a compression method combining difference coding and entropy coding.

In a third aspect of the present application, a data compression device is provided. The device includes: a receiving module, a determining module and a sending module. The receiving module is configured to receive messages from the first network device. The message includes a plurality of first data items, the corresponding relationship between the plurality of first data items and the plurality of records is one-to-one correspondence, and the plurality of first data items come from corresponding records respectively. The determining module is configured to determine a first compression strategy based on a plurality of first data items. The sending module is configured to send the first compression policy to the first network device.

In an implementation manner of the third aspect, the multiple first data items are compressed based on the second compression strategy, and the device further A decompression module is included, configured to decompress the plurality of first data items using a second compression strategy.

In an implementation manner of the third aspect, the determination module is configured to: determine the first compression policy based on the decompressed plurality of first data items when the condition is met.

In an implementation manner of the third aspect, the condition includes: the elapsed time period after determining the second compression strategy reaches or exceeds a preset time period.

In an implementation manner of the third aspect, the condition includes: the data types of the multiple data items used to determine the second compression policy are different from the data types of the multiple first data items.

In an implementation manner of the third aspect, the condition includes: the distribution characteristics of the plurality of data items used to determine the second compression strategy are different from the distribution characteristics of the plurality of first data items.

In an implementation manner of the third aspect, the sending module is configured to: when the first compression strategy is different from the second compression strategy, send the first compression strategy to the first network device.

In an implementation manner of the third aspect, the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second compression parameter.

In an implementation manner of the third aspect, the first compression algorithm or the second compression algorithm includes at least one of the following: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding and A compression method combining entropy coding, or a compression method combining difference coding and entropy coding.

In an implementation manner of the third aspect, the sending module is further configured to send the first compression strategy to the second network device, so that the second network device compresses the plurality of second data items based on the first compression strategy, and the plurality of first compression strategies The data type of the second data item is the same as the data type of the plurality of first data items.

In a fourth aspect of the present application, a data compression device is provided. The device includes: a sending module and a receiving module. The sending module is configured to send messages to the management device. The message includes a plurality of data items, and the corresponding relationship between the plurality of data items and the plurality of records is one-to-one correspondence, and the plurality of data items come from corresponding records respectively. The receiving module is configured to receive the first compression policy from the management device. A first compression strategy is determined based on a plurality of data items.

In an embodiment of the fourth aspect, the device further includes a compression module. The compression module is configured to compress the data item using a second compression strategy, wherein the message includes the compressed data item.

In an implementation manner of the fourth aspect, the receiving module is further configured to receive the second compression policy from the management device.

In an implementation manner of the fourth aspect, the device further includes a determining module. The determining module is configured to determine a second compression strategy. The sending module is also configured to send the second compression policy to the management device.

In an implementation manner of the fourth aspect, the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second compression parameter.

In an implementation manner of the fourth aspect, the first compression algorithm or the second compression algorithm includes at least one of the following: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding and A compression method combining entropy coding, or a compression method combining difference coding and entropy coding.

In a fifth aspect of the present application, an electronic device is provided. The electronic device includes a processor and memory. The memory stores instructions for execution by the processor. When the instruction is executed by the processor, the electronic device is made to implement the operation according to the method in the above first aspect or any implementation manner thereof, or implement the operation according to the method in the above second aspect or any implementation manner thereof.

In a sixth aspect of the present application, a computer-readable storage medium is provided. Computer-executable instructions are stored on the computer-readable storage medium. When the computer-executable instructions are executed by the processor, the operation according to the method in the above first aspect or any implementation manner thereof may be realized, or the operation according to the method in the above second aspect or any implementation manner thereof shall be realized.

In a seventh aspect of the present application, a chip or a chip system is provided. The chip or chip system includes a processing circuit configured to perform operations according to the method in the above first aspect or any implementation thereof, or implement operations according to the method in the above second aspect or any implementation thereof.

In an eighth aspect of the present application, a computer program or computer program product is provided. The computer program or computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions which, when executed, implement operations according to the method in the first aspect or any of its implementations above, Or implement the operations according to the method in the above second aspect or any implementation manner thereof.

Description of drawings

FIG. 1 shows a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 2 shows a schematic diagram of multiple records of a network device provided by an embodiment of the present application;

Fig. 3 shows the signaling interaction diagram of the data compression process provided by the embodiment of the present application;

FIG. 4 shows a signaling interaction diagram of the data compression process provided by the embodiment of the present application;

FIG. 5 shows a schematic diagram of a data packet provided by an embodiment of the present application;

FIG. 6 shows a signaling interaction diagram of the data compression process provided by the embodiment of the present application;

FIG. 7 shows a schematic block diagram of a data compression device provided by an embodiment of the present application;

FIG. 8 shows a schematic block diagram of a data compression device provided by an embodiment of the present application; and

Fig. 9 shows a schematic block diagram of an example device that can be used to implement the embodiments of the present application.

Detailed ways

In a communication network, network devices such as distributed probes need to report various data, such as performance measurement data, operation logs, security event information, etc., to management devices such as cloud security centers. Network devices measure and report a large amount of data, so there is a high demand for bandwidth. In this way, on the one hand, the interaction cost between the network device and the management device is high; on the other hand, a large amount of data exchanged between the network device and the management device occupies more network resources, which may affect other services. Therefore, an effective solution is needed to compress the interaction data between devices.

In order to solve the above problems and other potential problems, the embodiment of the present application provides a data compression scheme, which uses the same compression strategy for the corresponding multiple data items in multiple records, and can fully consider the corresponding data items. Commonality, improve the compression rate, and thus reduce the required bandwidth and reduce costs.

FIG. 1 shows a schematic diagram of an application scenario 100 provided by an embodiment of the present application. The application scenario 100 includes a management device 110 , a network device 121 , a network device 122 and a network device 123 . To simplify the description, the network device 121 , the network device 122 and the network device 123 may be collectively referred to as the network device 120 .

The "network device" in the embodiments of the present application may refer to a device that collects or measures data such as network security and network traffic in a computer network. Network devices can be implemented in various forms, or network devices can also have other names, including but not limited to, gateway devices, routers, network edge devices, firewall devices, network security probe devices, network probes, network measurement devices , detection equipment, etc.

In some scenarios, multiple network devices can belong to the same LAN, such as a campus network. For example, it may be assumed that network device 121 and network device 122 belong to the same campus network, and it may be assumed that network device 123 belongs to another campus network.

The "management device" in the embodiments of the present application may refer to a device that manages, summarizes and analyzes data such as network security and network traffic in a computer network. The management device can be implemented in various forms, or the management device can also have Other titles include, but are not limited to, servers, network security servers, security cloud platforms, traffic control devices, control centers, centralized data centers, management nodes, etc.

The network device 120 may send data to the management device 110, for example, the data may include security logs, forensic files, traffic logs, operation logs, alarm logs, and the like. Optionally, various logs may be recorded text in a specific format. Multiple records can be included in various logs. Each record can correspond to a different moment. Alternatively, each record can correspond to a different source. For example, the first record is an alarm record from network interface one, and the second record is an alarm record from network interface two.

Exemplarily, the network device 120 may obtain multiple records through measurement. For example, the network device 120 may measure periodically, for example, measure 150 records per second. For example, the network device 120 may periodically send the measured records to the management device 110, for example, once every second. For example, the network device 120 may send based on the number of records reaching a threshold, for example, send once every 200 records are measured. It can be understood that the network device 120 may also perform measurement and transmission based on other criteria, which will not be listed in this application.

Exemplarily, the network device may also obtain multiple records in other ways. For example, the network device 121 is also connected to the network device 131 and the network device 132 (both not shown in the figure), the network device 121 can receive the data sent by the network device 131 and the network device 132, and generate multiple records based on the received data . For example, each record may correspond to a different network device. For example, the first record is a security event record from the network device 131 , and the second record is not from the security event record of the network device 132 .

FIG. 2 shows a schematic diagram of multiple records 200 of the network device 120 provided by the embodiment of the present application. For illustration, five records are listed in FIG. 2 , for example, one row belongs to one record.

As shown in Figure 2, each record can include multiple data items, and different data items in the same record can represent different meanings, for example, the meanings represented by multiple data items in the same record in Figure 2 are from left to right For: metadata version, metadata identifier (identifier, ID), source Internet Protocol (Internet Protocol, IP) address, destination IP address, source transmission port, destination transmission port, protocol ID, byte increase count, packet increase count, etc. . It can be understood that at least two data items in the same record belong to different data types. For example, referring to FIG. The data type of is character.

Optionally, each data item in the same record belongs to different data types, or optionally, some data items in the same record (such as the source IP address and destination IP address in Figure 2) belong to the same data type.

As shown in Figure 2, different records may include data items with the same meaning, for example, a column in Figure 2 indicates the same meaning. It can be understood that data items in the same column may correspond to the same field and may have the same data type.

It can be understood that the form of rows and columns shown in FIG. 2 is only for illustration. In an actual scenario, multiple records can be formed by sequentially connecting multiple records, for example, different records are passed through specific symbols (such as &&, etc.) Separation, etc., for example, different records are placed inside different symbols (such as {}), etc., and different data items in the same record can be distinguished by different fields. This application is not limited to this.

It should be noted that the embodiment of the present application does not limit the transmission protocol between the network device 121 and the management device 110, for example, it may be a network configuration protocol (Network Configuration Protocol, NETCONF), a simple network management protocol (Simple Network Management Protocol, SNMP) , or a transmission mode determined through negotiation between the network device 121 and the management device 110 , and the like.

It should be noted that although three network devices are shown in FIG. 1 , the embodiments of the present application are not limited thereto, and in actual scenarios, more or fewer network devices may be included. As an illustration, the following embodiments mainly use the network device 121 as an example to illustrate the process of data compression. It can be understood that the embodiments of the present application are also applicable to other network devices. No more details in the text.

FIG. 3 shows a signaling interaction diagram of a data compression process 300 provided by an embodiment of the present application. Process 300 involves management device 110 and network device 121 .

The network device 121 sends 310 a message to the management device 110, the message including a plurality of first data items.

Exemplarily, the network device 121 may obtain multiple records through measurement, and each record includes multiple data items, for example, a first data item and a second data item. The correspondence between the multiple first data items and the multiple records is a one-to-one correspondence. That is, the multiple first data items in the message respectively come from corresponding records. In some examples, the plurality of first data items can be understood as a column as shown in FIG. 2 . In other examples, multiple first data items may be understood as the same field in different records.

Optionally, in some embodiments, the message may include a plurality of uncompressed first data items.

Optionally, in some embodiments, the network device 121 may determine a second compression strategy, and may use the second compression strategy to compress multiple first data items. Exemplarily, the message includes a compressed plurality of first data items.

For example, the second compression strategy may be determined by the network device 121 based on multiple first data items. Optionally, the network device 121 may also send the determined second compression policy to the management device 110 .

For example, the second compression strategy may be predefined, for example, called a default compression strategy or a preset compression strategy.

Optionally, as shown in FIG. 3 , the management device 110 may determine 302 the second compression policy, and send 304 the second compression policy to the network device 121 . Accordingly, network device 121 may compress 306 the plurality of first data items using the second compression strategy. It can be understood that the message includes a plurality of compressed first data items. Exemplarily, the management device 110 may determine the second compression strategy based on multiple previous data items, and the determination method is similar to the following method of determining the first compression strategy, which will not be repeated here.

Exemplarily, if the message includes a plurality of compressed first data items, the management device 110 may obtain 312 the plurality of first data items through decompression using the second compression strategy.

The management device 110 determines 320 a first compression strategy based on the plurality of first data items. Exemplarily, the management device 110 may determine the first compression policy based on the decompressed multiple first data items.

In the embodiment of the present application, a compression strategy (such as a first compression strategy or a second compression strategy) may include a compression algorithm and a compression parameter. The compression algorithm can include any of the following: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding combined with entropy encoding, or difference encoding combined with entropy encoding the compression method. The compression parameters correspond to the compression algorithms, and the compression parameters used by different compression algorithms are generally different.

In some embodiments, the management device 110 may determine whether the preset condition is met, and may determine the first compression strategy when it is determined that the preset condition is met.

Optionally, the preset condition may include that the time elapsed after the second compression policy is determined reaches or exceeds the preset time length. The preset period of time may be referred to as an update period, and in this manner, the management device 110 can periodically update the compression policy. As an example, the preset duration may be 1 second or other values. As an example, the management device 110 may determine the first compression strategy based on multiple first data items in all or part of the messages received within the update period, so that the determined first compression strategy can be more accurate.

Optionally, the preset condition may include that the data types of the multiple data items used to determine the second compression policy are different from the data types of the multiple first data items. For example, the data types of the plurality of data items used to determine the second compression strategy are integers (int), and the data types of the plurality of first data items are floating point types (float). In this way, the management device 110 can update the compression policy when the data type changes, so that the used compression policy can be more associated with the type of the compressed data, and the compression rate can be improved.

Optionally, the preset condition may include that the distribution characteristics of the multiple data items used to determine the second compression policy are different from multiple The distribution characteristics of the first data item. For example, the distribution characteristic of the plurality of data items used to determine the second compression strategy is sparse, that is, the number of empty data items among the plurality of data items here is large, while the distribution characteristic of the plurality of first data items is dense , that is, the number of data items that are empty among the plurality of first data items is small, wherein the large number may be greater than the first ratio threshold (such as 60%), and the small number may be smaller than the second ratio threshold (such as 20%) %), the second ratio threshold is less than or equal to the first ratio threshold. In this way, the management device 110 can update the compression strategy when the distribution characteristics of the data change, so that the used compression strategy can be more associated with the distribution characteristics of the compressed data, and the compression rate can be improved.

In the embodiment of the present application, the management device 110 may determine the first compression strategy based on data types, distribution characteristics, and the like of the multiple first data items. For example, the plurality of first data items are character strings and are distributed densely. Optionally, it may be determined that the first compression strategy includes a dictionary encoding compression manner. Optionally, the management device 110 may construct a neural network model to determine compression parameters of the first compression strategy, such as a dictionary.

The management device 110 sends 330 the first compression policy to the network device 121 . Optionally, when the first compression policy is different from the second compression policy, the management device 110 sends the first compression policy. In this way, repeated sending of the same compression strategy can be avoided, resulting in waste of transmission resources.

Exemplarily, the first compression strategy includes a first compression algorithm and first compression parameters, and the second compression strategy includes a second compression algorithm and second compression parameters. The difference between the first compression strategy and the second compression strategy may include that the first compression algorithm is different from the second compression algorithm, or may include that the first compression algorithm is the same as the second compression algorithm but that the first compression parameters are different from the second compression parameters, or It may include that the first compression algorithm is different from the second compression algorithm and that the first compression parameters are different from the second compression parameters.

It can be understood that after that, the network device 121 can use the first compression strategy to compress subsequent data items, so that a greater compression rate can be achieved based on the characteristics of the data items in the case of lossless compression.

In this way, the first compression strategy can be determined by the management device 110 , so that the network device 121 can compress data items using the first compression strategy determined by the management device 110 . This solution can make full use of the powerful processing capability of the management device 110, and use the same compression strategy for multiple data items of the same data type or multiple data items with the same distribution characteristics, which can improve the compression rate and reduce bandwidth occupation.

Some more detailed embodiments of the present application will be described below with reference to FIG. 4 to FIG. 6 .

FIG. 4 shows a signaling interaction diagram of a data compression process 400 provided by an embodiment of the present application. Process 400 involves management device 110 and network device 121 .

The network device 121 obtains more than 410 records through measurement. As an example, assume that the multiple records are N1 records, and N1 is a positive integer. For example, N1 is 25 or 200 or other values, which is not limited in the present application.

Each record can include multiple data items, each representing a different meaning. Optionally, as an illustration, multiple records may be represented in the form of a list, including rows and columns as shown in FIG. 2 , where a row represents one record, and the same column represents the same meaning.

In some embodiments, the plurality of records may be obtained through measurements after the last message sent by the network device 121 . In some embodiments, the plurality of records may be measured by the network device 121 after startup and before sending a message.

The network device 121 sends 420 a first message to the management device 110, the first message including a plurality of records. Correspondingly, the management device 110 may receive the first message from the network device 121 .

The embodiment of the present application does not limit the specific format of the message sent by the network device 121 to the management device 110, for example, it may be a log (log) file, for example, it may be in csv format.

For example, it may be assumed that the network device 121 sends the first message to the management device 110 at time t0.

In some embodiments, the network device 121 may send the uncompressed multiple records to the management device 110 through the first message. For example, the network device 121 may send an uncompressed first message when sending for the first time when it is just started. For example, a serialized data stream may be formed based on multiple records and sent to the management device 110 .

In some embodiments, the network device 121 may use a pre-configured compression policy to form the first message by compressing multiple records. For example, the network device 121 uses a pre-configured compression strategy when sending for the first time when it is just started. The pre-configured compression strategy may be pre-determined through negotiation between the network device 121 and the management device 110, or may be configured by the management device 110 to the network device 121, or may be based on a communication protocol between the network device 121 and the management device 110. Defined. The compression algorithm included in the pre-configured compression policy may be, for example, static dictionary encoding or the like. Correspondingly, the compression parameters included in the pre-configured compression strategy are, for example, dictionaries. Optionally, the preconfigured compression strategy may also be called a general compression strategy or other names, which is not limited in this application.

In some embodiments, the network device 121 may use the first compression strategy to form the first message by compressing multiple records. Exemplarily, the first compression policy may be the latest received by the network device 121 from the management device 110 before sending the first message. For example, the network device 121 receives the first compression policy from the management device 110 before sending for the first time at startup. For example, the network device 121 receives the first compression policy from the management device 110 after sending a message last time. For example, the network device 121 used the first compression strategy when sending a message last time. Optionally, the first message may also include an identifier of the first compression policy, such as a version number, a time stamp, and the like.

As a simplified example, it may be assumed that the first message is the first message about measured data that the network device 121 sends to the management device 110 after startup, then the first message includes uncompressed multiple records, or the first message includes a plurality of records compressed using a pre-configured compression strategy, or the first message includes a plurality of records compressed using a first compression strategy received after startup.

Exemplarily, the network device 121 can periodically send a message to the management device 110, assuming that the sending period is dt, then it can be understood that the network device 121 can send a message to the management device 110 at t0+dt, t0+2*dt, ... , for example dt is equal to 1 second or some other value.

The management device 110 determines 430 a second compression strategy based at least on the first message, the second compression strategy comprising at least one compression algorithm.

Optionally, in some embodiments, considering that multiple records are not compressed, or considering that the compression policy used by the first message is not determined based on the data of the network device 121, then the management device 110 may receive the After the first message after the network device 121 is powered on, the second compression policy is determined based on the first message.

Optionally, in some embodiments, considering the possible temporality/inaccuracy of multiple records, and also to improve the accuracy of the determined second compression strategy, the compression strategy may be determined periodically. For example, the second compression strategy may be determined based on messages (including the first message) received between time t0 and time t0+T. In other words, the management device 110 may determine the compression strategy based on the period T.

Exemplarily, the management device 110 may determine multiple records based on the first message, and determine the second compression policy based at least on the multiple records.

In some embodiments, the first message includes multiple uncompressed records, then the management device 110 can read the multiple records in the first message. In some embodiments, the first message includes the plurality of records compressed, for example using a pre-configured compression strategy or a first compression strategy. Then the management device 110 can use the corresponding compression strategy to perform decompression to obtain multiple records.

Exemplarily, each of the multiple records includes multiple data items, and different data items in the same record may represent different meanings. It may be assumed that the first message comprises N1 records, and eg each record comprises M1 data items.

Optionally, if the management device 110 determines the compression policy based on the period T, then the management device 110 may determine the second compression policy based on multiple messages (including the first message) received within the period T. For example, assuming that the number of multiple messages is K1, then K1*N1 records can optionally be determined based on the multiple messages.

In this embodiment of the present application, the management device 110 may determine the second compression policy based on the N records. For example, the N records are N1 records in the first message, or the N records may be a collection of multiple records in multiple messages.

In order to simplify the description, it can be assumed that N records are represented as shown in Figure 2, that is, one record can be represented as one row, different rows belong to different records, and data items with the same meaning in different records can be set to In the same column, different columns represent different meanings. The following describes an embodiment of determining a second compression policy in conjunction with multiple records as shown in FIG. 2 , but it should be understood that the embodiment of the present application may also be applied to other forms of records, which will not be listed here.

Exemplarily, the management device 110 may divide the column data of N records into multiple groups based on the attributes of the data items in each column. The attributes of a data item may include, but are not limited to: data type, sparseness of data distribution, range of values, and the like. The data type may include an integer type (int), a floating point type (float), a character string (string), and the like. The degree of sparseness of the data distribution can be determined based on whether there is a corresponding record that is NULL in the column, or the number of consecutive NULL records, and the like. When the data item is a numerical value, the range of the numerical value can be the difference between the maximum value and the minimum value of the value in the column, the difference between the maximum value and the minimum value of the absolute value of the value, etc.

For example, FIG. 5 shows a simplified diagram of multiple records 500. In FIG. 5, one row represents one record, and each record includes multiple columns: column A, column B, column C, . . . . Assuming that the data types of the data items in column A, column B, and column E are character strings, column A, column B, and column E can be divided into the same group, such as group 510 . Assuming that the data types of the data items in column C, column G and column H are integer, and the distribution of values is dense, then column C, column G and column H can be divided into the same group, such as group 520 . Assuming that the data types of the data items in column D and column F are integer, and the distribution of values is sparse, then column D and column F can be divided into the same group, such as group 530 .

Optionally, the management device 110 may perform cluster analysis on the data items in each column, statistically learn the correlation between the columns, and divide the multiple columns into multiple groups based on the correlation. Exemplarily, the cluster analysis may adopt an unsupervised learning method, a supervised classification method, or a predefined classification method, etc., which is not limited in the present application. It can be understood that only one column may be included in a certain group, or multiple columns may be included in a certain group.

Exemplarily, the management device 110 determines corresponding compression algorithms for each group. Accordingly, the compression parameters used by the compression algorithm can be determined.

In the embodiment of the present application, the compression algorithm may include but not limited to: dictionary encoding compression, difference encoding compression, run-length encoding compression, entropy encoding compression, dictionary encoding and entropy encoding combined compression, or difference encoding Compression combined with entropy coding.

For example, for numerical values, if the value changes frequently, run-length encoding compression can be used; if the value changes slowly, difference encoding compression can be used. For example, for strings, you can use dictionary encoding compression. It should be understood that this example is only for illustration, and does not constitute a limitation to the embodiment of the present application.

As an example, with reference to FIG. 5 , it can be assumed that the first compression algorithm is determined for group 510, the second compression algorithm is determined for group 520, and the third compression algorithm is determined for group 530, wherein the first compression algorithm, the second compression algorithm, the third Any two of the compression algorithms may be the same or different. For example, it is assumed that the first compression algorithm is a combination of dictionary coding and entropy coding, the second compression algorithm is a compression method of difference coding and entropy coding, and the third compression algorithm is a run-length coding compression method.

Optionally, the management device 110 may also determine compression parameters used by each compression algorithm. For example, the entropy coding table used in the entropy coding compression method can be determined, for example, the dictionary used in the dictionary coding compression method can be determined, for example, it can be determined Define the difference upper limit threshold used by the difference encoding compression method, and so on.

For example, referring to FIG. 5 , the entropy encoding table corresponding to the group 510 can be determined based on the occurrence probability of each character string in each column in the group 510 (ie, column A, column B, and column E). The entropy encoding table corresponding to the group 520 can be determined based on the distribution probability of each number in each column of the group 520 (ie, column C, column G, and column H), etc. The dictionary corresponding to the group 510 can be determined based on the frequency of occurrence of each character string in each column (i.e., column A, column B, and column E) in the group 510, for example, a neural network model (such as a stateful deep network model) can be used to determine Extract the high-frequency strings (also known as words in the dictionary) of the data items in each column to create different dictionaries for each column. Optionally, the neural network model used to extract high-frequency words may be obtained by multiple network devices 120 through federated learning and other methods. In this way, the strong computing capability of the management device 110 can be fully utilized, and the dictionary can be determined by the management device 110 , avoiding the waste of time and storage space resources caused by the network device 121 searching for matching algorithms.

In some embodiments, the management device 110 may respectively construct a neural network model for each group, and determine a compression algorithm based on the constructed neural network model. Exemplarily, the neural network model in the embodiment of the present application may be obtained by the management device 110 instructing multiple network devices 120 through federated learning, incremental learning, migration learning, etc., so that more measurement data can be considered comprehensively, The speed of model construction is improved, and the difference in measurement data of each network device 120 can be fully considered, so that the obtained neural network model can be made more accurate.

In this way, the management device 110 can determine the compression strategy at the granularity of the columns. For example, the run-length encoding compression method can be used for sparsely distributed columns, and the compression method combining difference coding and entropy coding can be used for columns with frequent numerical changes. Columns of string type can be compressed by combining dictionary encoding (such as static dictionary encoding and sliding window dictionary encoding) with entropy encoding, and so on. Compared with taking records (rows) as the granularity, it is possible to determine the corresponding compression strategy for the same type of data, thereby compressing the redundancy in the column direction and improving the compression rate.

Exemplarily, the static dictionary encoding can adjust the compression effect by changing the size of the static dictionary, the arrangement order of high-frequency words in the dictionary, and the like. Exemplarily, the sliding window dictionary encoding can adjust the compression effect by changing the size of the sliding window. It should be noted that although some possible compression algorithms are listed above, the present application does not limit them. For example, compression algorithms may also include Huffman coding, arithmetic coding, interval coding, and the like. Exemplarily, the Huffman coding can construct a Huffman tree according to the frequency of symbols, and adjust the compression effect by rebuilding the Huffman tree. The arithmetic coding can calculate the probability of the symbol string according to the probability of the symbol occurrence, obtain a parameter greater than 0 and less than 1, and recalculate the probability of the symbol string according to the probability of the symbol occurrence to adjust the compression effect. Interval encoding can map a symbol string to a small interval in a large interval value according to the probability of symbol occurrence, and recalculate the symbol string probability according to the probability of symbol occurrence to adjust the compression effect. Regarding various compression algorithms, reference may be made to existing technologies, which will not be listed one by one in this application.

The management device 110 sends 440 the second compression policy to the network device 121 . Optionally, the management device 110 may also send the identifier of the second compression policy to the network device 121, such as the version number of the second compression policy, a timestamp, or a sequence obtained by scrambling the timestamp through a predefined hash sequence wait. Optionally, when the second compression policy is different from the first compression policy, the second compression policy is sent.

In the embodiment of the present application, considering the powerful processing capability at the management device 110, the time-consuming time for the management device 110 to determine the second compression strategy can be ignored, and it can be assumed that the management device 110 determines and sends the second compression strategy almost simultaneously, For example, assume time t1.

As an example, it may be assumed that the version number of the second compression policy is V1, and the time stamp is t1.

The second compression strategy may include at least one compression algorithm. Referring to FIG. 5 , the second compression strategy may include grouping methods and corresponding compression algorithms for each group, as shown in Table 1 below.

Table 1: Second Compression Strategy

Correspondingly, it can be understood that the network device 121 may receive the second compression policy from the management device 110 at time t1.

Exemplarily, after receiving the second compression policy, the network device 121 may use the second compression policy to compress 450 the data, and send 460 the second message to the management device 110 . Correspondingly, the management device 110 receives the second message from the network device 121 .

Specifically, the network device 121 may divide the multi-column data items into multiple groups according to the second compression strategy, and perform compression using a corresponding compression algorithm for each group.

Optionally, the second message may further include an identifier of the second compression policy (such as a version number V1), so that the management device 110 determines how to decompress. In this way, it is possible to avoid data decompression errors caused by the inconsistency of the latest versions of the compression policies at the network device 121 and the management device 110 .

It can be understood that if the network device 121 fails to receive the second compression policy sent by the management device 110 at time t1, then the message sent by the network device 121 to the management device 110 after time t1 is different from the first message sent by the aforementioned process 420 Similarly, to avoid repetition, details will not be repeated here. For the management device 110, after sending the second compression strategy, if it finds that the network device 121 does not use the second compression strategy for compression, it may resend the second compression strategy to the network device 121 after a retransmission time period. For example, assuming that the retransmission duration is T0, the second compression strategy may be retransmitted at time t1+T0. Optionally, the maximum number of retransmissions can be preset, such as nr, then after t1+nr*T0, even if the network device 121 still has not received the second compression policy, the management device 110 will not retransmit the second compression policy. Compression strategy.

The management device 110 may decompress 470 the second message based on the second compression policy.

The management device 110 determines 480 a third compression policy. Exemplarily, the management device 110 may determine whether the preset condition is met, and may determine the third compression strategy when it is determined that the preset condition is met. In this way, it is possible to update the compression strategy, specifically to update the second compression strategy to the third compression strategy, so as to better adapt to changes in the data measured by the network device 121 .

In some embodiments, the preset condition may include: the time elapsed after the second compression strategy is determined reaches or exceeds a preset time length. For example, assuming that the preset duration is T, then if the current moment>=t1+T, the third compression strategy may be determined. Optionally, the preset time length (that is, T) may be referred to as a policy update period. In this way, the compression policy can be updated periodically, so that it can be ensured that the used compression policy corresponds to the data policyd by the network device 121 , and lossless compression of data can be realized.

In some embodiments, the preset condition may include: the data type and/or distribution feature of each column of data items in the message from the network device 121 changes. For example, in the example in FIG. 5 , the data items in column D are of sparsely distributed integer type, if after time t0, the data items in column D become densely distributed and their values change more frequently. Then the third compression strategy can be re-determined based on this. For example, if the data types and distribution characteristics of the data items in the remaining columns are basically unchanged, you can determine to regroup, divide column A, column B, and column E into one group, and divide column C, column D, column G, and column H is divided into one group, column F is divided into one group; and then the compression algorithm can be determined separately for each group. In this way, the compression strategy can be updated based on the change of the data item, so that the online serialization compression of the measured data by the network device 121 can guarantee performance and realize lossless compression.

Optionally, for the manner in which the management device 110 determines the third compression policy, reference may be made to the embodiment of determining the second compression policy in the above-mentioned process 430 , and details are not repeated here for brevity.

The management device 110 sends 490 the third compression policy to the network device 121 . Optionally, the management device 110 may also send the identifier of the third compression policy to the network device 121, such as the version number and time stamp of the third compression policy. As an example, it may be assumed that the version number of the third compression policy is V2, and the time stamp is t2.

Exemplarily, the further process after process 490 is similar to the process after process 440 above, which will not be repeated in this application.

It should be noted that, in order to simplify the description, the transmission delay between the network device 121 and the management device 110 is not considered in the above embodiments, but in actual scenarios, there is a transmission delay between the two, for example, Δt. For example, the network device 121 sends the first message at time t0, and accordingly the management device 110 receives the first message at time t0+Δt. The transmission delays of other information between the network device 121 and the management device 110 are similar, and will not be described in detail here.

It can be understood that although the data compression process for the network device 121 is only described in connection with the process 400 in FIG. 4 , this process is also applicable to other network devices. For example, for each network device 120, the management device 110 may respectively determine the compression policy of each network device. In this way, management device 110 is able to determine compression policies for individual network devices 120 . Correspondingly, the management device 110 can obtain local (for example, at the granularity of the campus LAN) or global (for example, all network devices 120 interacting with the management device 110) statistical information of the compression policy, for example, the compression policy of each version, determine the compression policy The neural network model used, the history of the compression strategy being used, etc.

Optionally, in some embodiments, for another network device restarted or powered on, taking network device 122 as an example, before receiving a message from network device 122, management device 110 may send a compressed Strategy. For example, if the network device 122 starts between time t1 and t2, and the network device 122 and the network device 121 belong to the same local area network, the management device 110 can set the latest compression policy for the network device 121 (as mentioned above at time t1 to The determined second compression strategy) is sent to the network device 122. For example, the management device 110 may determine the correlation between the network device 122 and the network device 121, and when the correlation is greater than a correlation coefficient threshold (such as 80% or other values), the second compression strategy determined for the network device 121 sent to the network device 122, wherein the correlation may be determined based on device attributes, which may include but not limited to the IP address of the device, the physical address of the device, the historical data of the device, the type of the device, and the like.

For example, the management device 110 may send the compression policy with the longest historical usage time to the network device 122 based on the statistical information. For example, the management device 110 may send the compression policy with the highest compression ratio to the network device 122 . In this way, a newly started network device (such as the network device 122 ) can perform data compression based on the instruction of the management device 110 , avoiding large bandwidth occupation caused by uncompressed data.

In this way, the management device in the embodiment of the present application determines the compression strategy based on the multiple data items corresponding to the multiple records, and the corresponding multiple data items have the same data type or the same distribution characteristics. Therefore, the management device is more able to Consider the redundancy between data items of the same data type or data items of the same distribution characteristics, so as to make the compression rate higher, which can make the compressed data occupy less bandwidth and save costs. Moreover, the management device can update the compression strategy based on the update period or based on the change of the data item, so as to ensure that the used compression strategy corresponds to the data item and ensure the lossless compression of the data.

Fig. 6 shows a signaling interaction diagram of a data compression process 600 according to some embodiments of the present application. Process 600 involves management device 110 and network device 121 .

The network device 121 obtains more than 610 records through measurement. As an example, assume that the number of records is N1 records. For multiple records, refer to the aforementioned process 410 in conjunction with FIG. 4 and the related description in conjunction with FIG. 2. For example, multiple records can be represented as multiple columns, and multiple data items in each column can have the same data type. For brevity, I won't go into details here.

The network device 121 determines 620 a first compression policy based on the plurality of records. The first compression strategy may include a first compression algorithm method and the first compression parameter.

Exemplarily, the network device 121 may divide multiple records into multiple groups at column granularity, and each group may include one or more columns. For each of the multiple groups, the network device 121 may separately determine a compression algorithm, for example, different groups may use different compression algorithms and/or compression parameters.

Specifically, the process for the network device 121 to determine the first compression policy is similar to the process 430 for the management device 110 to determine the second compression policy described above in conjunction with FIG. Do not repeat.

The network device 121 compresses the plurality of records using a first compression strategy to generate 630 a first message. The first message may include multiple compressed records, and the multiple records are compressed at the column granularity, that is, the same column in the multiple records is compressed using the same compression algorithm and compression parameters.

The network device 121 sends 642 the first message to the management device 110 . The network device 121 sends 644 the first compression policy to the management device 110 .

Optionally, the first message and the first compression policy may be carried in the same signaling and sent, or may be sent separately, which is not limited in this application. Optionally, the sent first compression policy may also include an identifier of the first compression policy, such as a version number, a time stamp, and the like.

The management device 110 uses the first compression policy to decompress 650 the first message to obtain multiple records.

It can be understood that the network device 121 may continue to perform data measurement, perform compression using the first compression strategy, and further send the compressed record to the management device 110 . Exemplarily, the network device 121 may perform sending according to a reporting period. Optionally, each sending may carry the identifier of the first compression policy used, so that the management device 110 can decompress successfully and correctly.

Optionally, as shown in FIG. 6 , the network device 121 may determine 660 a second compression policy. In some embodiments of the present application, the network device 121 may determine the second compression strategy when it is determined that the preset condition is met.

As an example, the preset condition may be that the update duration is reached or exceeded, for example, the time elapsed after the first compression strategy is determined reaches or exceeds the update duration. That is to say, the network device 121 may periodically update the compression policy according to the update period.

As another example, the preset condition may be that the data type or distribution characteristics of the data items in the record have changed. For example, in multiple consecutive reporting periods, the data type or distribution feature of one or more columns of data items changes. That is, network device 121 may update the compression policy based on changes in characteristics of data items.

In some embodiments, the network device 121 may determine the second compression policy based on each column of data items in multiple records in multiple reporting periods. For example, the second compression strategy may be determined based on the column data items in all or part of the records within the update duration after the first compression strategy is determined. It can be understood that the process of determining the second compression strategy is similar to the aforementioned process of determining the first compression strategy, and will not be described in detail here.

Exemplarily, the network device 121 may use the second compression strategy to compress the records obtained by the subsequent measurement, and send the records to the management device 110 . The network device 121 may also send the second compression policy to the management device 110, so that the management device 110 can successfully decompress.

In this way, in the embodiment of the present application, the network device 121 can determine the compression strategy based on the same type of data items, so that a more accurate compression strategy can be obtained, so that the compression rate is greater, and thus bandwidth occupation can be reduced.

It should be noted that although the above embodiments are described in conjunction with FIG. 4 and FIG. 6 in which the compression strategy is determined by the management device 110 and the compression strategy is determined by the network device 121, the above embodiments are not independent of each other, and they can also be Combine to get other examples. For example, in the process 600 of FIG. 6 , the management device 110 may also send the The first compression policy or the second compression policy is sent to another network device 122 and so on. The embodiments of the present application do not list various combinations.

It should be understood that in the embodiments of the present application, expressions such as "first", "second", and "third" are only used to indicate that multiple objects may be different, but at the same time, it does not rule out that the two objects are the same . Expressions such as "first", "second", and "third" should not be construed as limiting the embodiments of the present application.

It should also be understood that the divisions of methods, situations, categories and embodiments in the embodiments of the present application are only for the convenience of description, and should not constitute special limitations. Logical cases can be combined with each other.

It should also be understood that the foregoing content is only intended to help those skilled in the art better understand the embodiments of the present application, and is not intended to limit the scope of the embodiments of the present application. Those skilled in the art may make various modifications or changes or combinations based on the above contents. Such modifications, changes or combined solutions are also within the scope of the embodiments of the present application.

It should also be understood that the above description focuses on the differences between the various embodiments, and the same or similar points may refer to or learn from each other, and for the sake of brevity, details are not repeated here.

Fig. 7 shows a schematic block diagram of an apparatus 700 provided by an embodiment of the present application. Apparatus 700 may be implemented by software, hardware or a combination of both. In some embodiments, the apparatus 700 may be implemented as the management device 110 shown in FIG. 1 or as a chip in the management device 110 .

As shown in FIG. 7 , the apparatus 700 includes a receiving module 710 , a determining module 720 and a sending module 730 . Optionally, the device 700 may further include a decompression module 740 .

The receiving module 710 is configured to receive a message from the first network device, the message includes a plurality of first data items, the correspondence relationship between the plurality of first data items and the plurality of records is one-to-one correspondence, and the plurality of first data items are from to the corresponding record. The determining module 720 is configured to determine a first compression strategy based on the plurality of first data items. The sending module 730 is configured to send the first compression policy to the first network device.

In some embodiments, the plurality of first data items are compressed based on a second compression strategy. The decompression module 740 may be configured to decompress the plurality of first data items using a second compression strategy.

Exemplarily, the determining module 720 may be configured to determine a first compression policy based on the decompressed multiple first data items when a condition is met. Optionally, the condition includes that the time elapsed after the second compression strategy is determined reaches or exceeds a preset time length. Optionally, the condition includes that the data types of the plurality of data items used to determine the second compression policy are different from the data types of the plurality of first data items. Optionally, the condition includes that the distribution characteristics of the plurality of data items used to determine the second compression policy are different from the distribution characteristics of the plurality of first data items.

Specifically, the sending module 730 may be configured to send the first compression strategy to the first network device when the first compression strategy is different from the second compression strategy.

In some embodiments, the first compression strategy includes a first compression algorithm and first compression parameters, and the second compression strategy includes a second compression algorithm and second compression parameters. The first compression algorithm or the second compression algorithm includes at least one of the following: dictionary coding compression, difference coding compression, run-length coding compression, entropy coding compression, dictionary coding and entropy coding combined compression, or difference coding Compression combined with entropy coding.

Optionally, the sending module 730 may also be configured to send the first compression strategy to the second network device, so that the second network device compresses multiple second data items based on the first compression strategy. The data type of the plurality of second data items is the same as the data type of the plurality of first data items.

The apparatus 700 in FIG. 7 can be used to implement the processes described above in conjunction with the management device 110 in FIG. 3 to FIG. 6 , and details are not repeated here for the sake of brevity.

Fig. 8 shows a schematic block diagram of an apparatus 800 according to some embodiments of the present application. Apparatus 800 may be implemented by software, hardware or a combination of both. In some embodiments, the apparatus 800 may be implemented as the network device 120 shown in FIG. 1 or as a chip in the network device 120 .

As shown in FIG. 8 , the device 800 includes a sending module 810 and a receiving module 820 . Optionally, the apparatus 800 may further include a compression module 830 and/or a determination module 840 .

The sending module 810 is configured to send a message to the management device, the message includes multiple data items, and the correspondence between the multiple data items and the multiple records is a one-to-one correspondence, and the multiple data items come from corresponding records. The receiving module 820 is configured to receive a first compression strategy from the management device, the first compression strategy is determined based on a plurality of data items.

In some embodiments, the apparatus 800 also includes a compression module 830 . The compression module 830 is configured to compress the data item using a second compression strategy, wherein the message includes the compressed data item.

Optionally, the receiving module 820 is further configured to receive the second compression policy from the management device. Or optionally, the determining module 840 may be configured to determine the second compression policy, and the sending module 810 is further configured to send the second compression policy to the management device.

Exemplarily, the first compression strategy includes a first compression algorithm and first compression parameters, and the second compression strategy includes a second compression algorithm and second compression parameters. The first compression algorithm or the second compression algorithm includes at least one of the following: dictionary coding compression, difference coding compression, run-length coding compression, entropy coding compression, dictionary coding and entropy coding combined compression, or difference coding Compression combined with entropy coding.

The apparatus 800 in FIG. 8 can be used to implement the various processes described above in conjunction with the network device 120 (including the network device 121 ) in FIGS. 3 to 6 , and details are not repeated here for brevity.

The division of modules or units in the embodiments of the present application is schematic, and it is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional unit in the disclosed embodiments can be integrated In one unit, it may exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Fig. 9 shows a schematic block diagram of an example device 900 that may be used to implement embodiments of the present application. The device 900 may be implemented as or included in the management device 110 or the network device 120 shown in FIG. 1 .

As shown in the figure, the device 900 includes a central processing unit (Central Processing Unit, CPU) 901, a read-only memory (Read-Only Memory, ROM) 902, and a random access memory (Random Access Memory, RAM) 903. The CPU 901 can perform various appropriate actions and processes according to computer program instructions stored in the ROM 902 and/or RAM 903 or loaded from the storage unit 908 into the ROM 902 and/or RAM 903. In the ROM 902 and/or the RAM 903, various programs and data necessary for the operation of the device 900 can also be stored. The CPU 901 and the ROM 902 and/or RAM 903 are connected to each other via a bus 904. An input/output (Input/Output, I/O) interface 905 is also connected to the bus 904 .

Multiple components in the device 900 are connected to the I/O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc. ; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the device 900 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

CPU 901 may be various general and/or special purpose processing components having processing and computing capabilities. Some examples that can be implemented include but are not limited to Graphics Processing Unit (Graphics Processing Unit, GPU), various dedicated artificial intelligence (Artificial Intelligence, AI) computing chips, various computing units that run machine learning model algorithms, digital signal Processor (Digital Signal Processor, DSP), and any appropriate processor, controller, microcontroller, etc., relative Accordingly, it can be called a computing unit. The CPU 901 executes the various methods and processes described above. For example, in some embodiments, the processes described above may be implemented as a computer software program tangibly embodied on a computer-readable medium, such as storage unit 908 . In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 900 via the ROM 902 and/or RAM 903 and/or the communication unit 909 . When a computer program is loaded into ROM 902 and/or RAM 903 and executed by CPU 901, one or more steps of the process described above may be performed. Alternatively, in other embodiments, the CPU 901 may be configured in any other suitable manner (for example, by means of firmware) to execute the processes described above.

Exemplarily, the device 900 in FIG. 9 may be implemented as a computing device, or may be implemented as a chip or a chip system in the computing device, which is not limited by the embodiments of the present application.

Embodiments of the present application also provide a chip, which may include an input interface, an output interface, and a processing circuit. In the embodiment of the present application, the interaction of signaling or data may be completed by the input interface and the output interface, and the generation and processing of signaling or data information may be completed by the processing circuit.

Embodiments of the present application also provide a chip system, including a processor, configured to support a computing device to implement the functions involved in any of the foregoing embodiments. In a possible design, the system-on-a-chip may further include a memory for storing necessary program instructions and data, and when the processor runs the program instructions, the device installed with the system-on-a-chip can implement the program described in any of the above-mentioned embodiments. Methods. Exemplarily, the chip system may consist of one or more chips, and may also include chips and other discrete devices.

Embodiments of the present application further provide a processor, configured to be coupled with a memory, where instructions are stored in the memory, and when the processor executes the instructions, the processor executes the methods and functions involved in any of the foregoing embodiments.

Embodiments of the present application also provide a computer program product including instructions, which, when run on a computer, enable the computer to execute the methods and functions involved in any one of the above embodiments.

Embodiments of the present application also provide a computer-readable storage medium, on which computer instructions are stored, and when a processor executes the instructions, the processor is made to execute the methods and functions involved in any of the above embodiments.

In general, the various embodiments of the present application may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor or other computing device. While various aspects of the embodiments of the present application are shown and described as block diagrams, flowcharts, or using some other pictorial representation, it should be understood that the blocks, devices, systems, techniques or methods described herein may be implemented as, without limitation, Exemplary, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof.

The present application also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product comprises computer-executable instructions, eg included in program modules, which are executed in a device on a real or virtual processor of a target to perform the process/method as above with reference to the accompanying drawings. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In various embodiments, the functionality of the program modules may be combined or divided as desired among the program modules. Machine-executable instructions for program modules may be executed within local or distributed devices. In a distributed device, program modules may be located in both local and remote storage media.

Computer program codes for implementing the methods of the present application may be written in one or more programming languages. These computer program codes can be provided to processors of general-purpose computers, special-purpose computers, or other programmable data processing devices, so that when the program codes are executed by the computer or other programmable data processing devices, The functions/operations specified in are implemented. The program code may reside entirely on the computer, partly on the computer, as a stand-alone software package, Execute partly on the computer and partly on the remote computer or entirely on the remote computer or server.

In the context of the present application, computer program code or related data may be carried by any suitable carrier, so that a device, apparatus or processor can perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

A computer readable medium may be any tangible medium that contains or stores a program for or related to an instruction execution system, apparatus, or device. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer diskettes, hard disks, random storage access memory (RAM), read-only memory (ROM), erasable programmable read-only Memory (Erasable Programmable Read-Only Memory, EPROM) or flash memory, optical storage device, magnetic storage device, or any suitable combination thereof.

In addition, while operations of the methods of the present application are depicted in a particular order in the figures, this does not require or imply that operations must be performed in that particular order, or that all illustrated operations must be performed, to achieve desirable results. Conversely, the steps depicted in the flowcharts may be performed in an altered order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present application may be embodied in one device. Conversely, the features and functions of one device described above may be further divided to be embodied by a plurality of devices.

Various implementations of the present application have been described above, and the above description is exemplary, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The choice of terminology used herein is intended to well explain the principles of each implementation, practical applications or improvements to technologies in the market, or to enable other ordinary skilled persons in the technical field to understand the various implementations disclosed herein.

Claims

A data compression method, characterized in that, comprising:

The management device receives a message from the first network device, the message includes a plurality of first data items, the correspondence between the plurality of first data items and the plurality of records is a one-to-one correspondence, and the plurality of first data items respectively from the corresponding records;

the management device determines a first compression policy based on the plurality of first data items; and

The management device sends the first compression policy to the first network device.
The method according to claim 1, wherein the plurality of first data items are compressed based on a second compression strategy, the method further comprising:

The management device uses the second compression strategy to decompress the plurality of first data items.
The method according to claim 2, wherein said management device determining a first compression policy based on said plurality of first data items comprises:

When a condition is met, the management device determines the first compression strategy based on the decompressed plurality of first data items.
The method according to claim 3, wherein the conditions include:

The elapsed time period after the management device determines the second compression policy reaches or exceeds a preset time period.
The method according to claim 3, wherein the conditions include:

The data types of the plurality of data items used to determine the second compression strategy are different from the data types of the plurality of first data items, or, the distribution characteristics of the plurality of data items used to determine the second compression strategy It is different from the distribution characteristics of the plurality of first data items.
The method according to any one of claims 2 to 5, wherein the sending the first compression policy by the management device to the first network device comprises:

When the first compression policy is different from the second compression policy, the management device sends the first compression policy to the first network device.
The method according to any one of claims 2 to 6, wherein the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second Compression parameters.
The method according to claim 7, wherein the first compression algorithm or the second compression algorithm comprises at least one of the following:

Dictionary encoding compression method,

Difference encoding compression method,

Run-length encoding compression method,

Entropy coding compression method,

A combination of dictionary encoding and entropy encoding, or

A compression method combining difference coding and entropy coding.
The method according to any one of claims 1 to 8, further comprising:

The management device sends the first compression strategy to the second network device, so that the second network device compresses a plurality of second data items based on the first compression strategy, and the data of the plurality of second data items The type is the same as the data type of the plurality of first data items.
A data compression method, characterized in that, comprising:

The network device sends a message to the management device, the message includes a plurality of data items, the correspondence relationship between the plurality of data items and the plurality of records is one-to-one correspondence, and the plurality of data items come from corresponding records respectively; and

The network device receives a first compression policy from the management device, the first compression policy being determined based on the plurality of data items.
The method according to claim 10, characterized in that the method further comprises:

The network device compresses the data item using a second compression strategy, wherein the message includes the compressed data item.
The method according to claim 11, further comprising:

The network device receives the second compression policy from the management device.
The method according to claim 11, characterized in that the method further comprises:

the network device determines the second compression policy; and

The network device sends the second compression policy to the management device.
The method according to any one of claims 11 to 13, wherein the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second Compression parameters.
A data compression device is characterized in that it comprises:

The receiving module is configured to receive a message from the first network device, the message includes a plurality of first data items, the correspondence between the plurality of first data items and the plurality of records is one-to-one correspondence, and the plurality of The first data items are respectively from the corresponding records;

a determining module configured to determine a first compression strategy based on the plurality of first data items; and

A sending module configured to send the first compression policy to the first network device.
The device according to claim 15, wherein the plurality of first data items are compressed based on a second compression strategy, and the device further comprises a decompression module configured to use the second compression strategy for The plurality of first data items are decompressed.
The device according to claim 16, wherein the determination module is configured to:

When a condition is met, the first compression strategy is determined based on the decompressed plurality of first data items.
The apparatus of claim 17, wherein the conditions include:

The elapsed time period after determining the second compression policy reaches or exceeds a preset time period.
The apparatus of claim 17, wherein the conditions include:

The data types of the plurality of data items used to determine the second compression strategy are different from the data types of the plurality of first data items, or, the distribution characteristics of the plurality of data items used to determine the second compression strategy It is different from the distribution characteristics of the plurality of first data items.
The device according to any one of claims 16 to 19, wherein the sending module is configured to:

When the first compression policy is different from the second compression policy, sending the first compression policy to the first network device.
The device according to any one of claims 16 to 20, wherein the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second Compression parameters.
The device according to any one of claims 15 to 21, wherein the sending module is further configured to send the first compression policy to a second network device, so that the second network device based on the The first compression strategy compresses a plurality of second data items, and the data type of the plurality of second data items is the same as the data type of the plurality of first data items.
A data compression device is characterized in that it comprises:

The sending module is configured to send a message to the management device, the message includes a plurality of data items, and the plurality of data items The correspondence relationship with multiple records is one-to-one correspondence, and the multiple data items are respectively from corresponding records; and

A receiving module configured to receive a first compression strategy from the management device, the first compression strategy is determined based on the plurality of data items.
The device according to claim 23, further comprising:

A compression module configured to compress the data item using a second compression strategy, wherein the message includes the compressed data item.
The apparatus according to claim 24, wherein the receiving module is further configured to receive the second compression policy from the management device.
The method according to claim 24, further comprising a determination module configured to determine the second compression strategy; and

The sending module is further configured to send the second compression policy to the management device.
The device according to any one of claims 24 to 26, wherein the first compression strategy includes a first compression algorithm and a first compression parameter, and the second compression strategy includes a second compression algorithm and a second Compression parameters.
An electronic device comprising a processor and a memory, on which computer instructions are stored, which when executed by the processor cause the electronic device to perform the method according to any one of claims 1 to 14 method.
A computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method according to any one of claims 1 to 14 is implemented.
A computer program product having embodied thereon computer executable instructions which, when executed, implement the method of any one of claims 1 to 14.