CN114968669A - Real-time data compression backup method and device and storage medium - Google Patents
Real-time data compression backup method and device and storage medium Download PDFInfo
- Publication number
- CN114968669A CN114968669A CN202210762078.1A CN202210762078A CN114968669A CN 114968669 A CN114968669 A CN 114968669A CN 202210762078 A CN202210762078 A CN 202210762078A CN 114968669 A CN114968669 A CN 114968669A
- Authority
- CN
- China
- Prior art keywords
- data
- compressed
- compression
- service data
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000013144 data compression Methods 0.000 title claims abstract description 31
- 238000007906 compression Methods 0.000 claims abstract description 118
- 230000006835 compression Effects 0.000 claims abstract description 118
- 238000010276 construction Methods 0.000 claims abstract description 55
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 12
- 238000005538 encapsulation Methods 0.000 claims description 14
- 230000015654 memory Effects 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 19
- 238000005457 optimization Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000002457 bidirectional effect Effects 0.000 description 2
- 230000006837 decompression Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a real-time data compression backup method, a real-time data compression backup device and a storage medium. In the invention, the service data to be backed up is transmitted to a compressed data construction module in a time sequence manner in a compression request manner to be divided into a plurality of sub-service data arranged in a time sequence manner; the compressed data construction module transmits the sub-service data to a compression engine, the sub-service data is compressed by the compression engine through a compression algorithm to obtain compressed sub-service data, and the compressed sub-service data is fed back to the compressed data construction module in a time sequence manner; the compressed data construction module arranges the received compressed sub-service data according to time sequence, and combines the compressed sub-service data with the compression control block and the filling zero into encapsulated data with fixed capacity; and the compressed data construction module writes the encapsulated data into a storage module of the backup server. The service data is divided, compressed and packaged in real time to obtain packaged data, the packaged data written into the disk are compressed data, the data volume is small, the writing speed is high, the resource occupation is small, and the efficiency is high.
Description
Technical Field
The present invention relates to the field of data real-time compression backup, and in particular, to a real-time data compression backup method, device and storage medium.
Background
With the development of social digitalization, the amount of data generated by various industries is increasing day by day, and especially the amount of multimedia data is very large. For example, the data size of a color photo can reach 10Mb, and the data size of video images and sound is more huge due to continuous playing, which causes great burden to the storage of the computer and network transmission. Data backup involves online, near-line, and offline storage for enterprise-level users. All the related storage modes need to consider efficiency optimization, especially optimization of an online storage process.
The storage efficiency optimization method provided by mainstream manufacturers in the current storage market mainly comprises the following steps: automatic hierarchical hot spot data migration, archive and space management, deduplication (typically used to archive or backup data), data compression, and the like. Most of these optimization methods are based on storage devices, studying how to place the already saved information more optimally, how to provide the host with these data or delete duplicate copies more efficiently. The data compression technology transforms data information, and less data are stored. However, most manufacturers in the storage industry currently adopt post-processing lossless compression, and in the compression method of post-processing, data needs to be acquired first, and more bandwidth storage resources of the storage device need to be occupied when the data is compressed, and the influence on the performance of the storage device is large.
Disclosure of Invention
In order to solve the technical problems or at least partially solve the technical problems, the invention provides a real-time data compression backup method, a real-time data compression backup device and a storage medium.
In a first aspect, the present invention provides a method for compressing and backing up real-time data, including: the method comprises the steps that service data needing to be backed up by a client side are transmitted to a compressed data construction module of a backup server in a time sequence mode in a compression request mode, and the compressed data construction module is used for segmenting the service data to form a plurality of sub-service data arranged in the time sequence;
the compressed data construction module transmits the sub-service data to a compression engine, the compression engine compresses each sub-service data through a compression algorithm to obtain compressed sub-service data, and the compressed sub-service data is fed back to the compressed data construction module in a time sequence;
the compressed data construction module arranges the received compressed sub-service data according to time sequence and combines the compressed sub-service data into encapsulated data with fixed capacity; wherein encapsulating the data comprises: sequentially arranged compressed sub-service data, a compression control block and zero padding; the zero padding is used for padding the encapsulated data to a fixed capacity, and the record content of the compression control block comprises a compression algorithm, the positions of the compressed sub-service data arranged according to the sequence of each compressed sub-service data and the real length of the compressed sub-service data;
and the compressed data construction module writes the encapsulated data into a storage module of the backup server.
Furthermore, communication is established between the client and the backup server through a compression session, the client encapsulates the service data into a compression request, and the compression request is sent to the backup server through the compression session;
the backup server will bind the compression session based on the compression instance implemented by the compression data construction module and the compression engine.
Further, each compression instance is configured with a first queue for queuing compression requests, the first queue providing compression requests in order for the compressed data construction module; each compression instance is provided with a second queue for queuing the sub-service data, and the second queue provides the sub-service data for the compression engine in order; each compression example is provided with a third queue for queuing the compressed sub-service data, and the third queue provides the compressed sub-service data for the compressed data construction module in order; the first queue, the second queue and the third queue are all FIFO queues.
Furthermore, the compressed data constructing module counts the total amount of the compressed sub-service data currently sorted, when the amount of the compressed sub-service data of the next time sequence is added to the total amount of the compressed sub-service data currently sorted and is greater than the fixed capacity of the encapsulated data, the fixed capacity subtracts the total amount of the compressed sub-service data currently sorted and the amount of the compressed control block to obtain the amount of the zero padding, and the compressed data constructing module encapsulates the compressed sub-service data currently sorted, the zero padding and the corresponding compressed control block to obtain the encapsulated data.
Furthermore, the compression control block is encapsulated at the tail part of the encapsulated data, and zero padding is arranged between the compressed sub-service data and the compression control block.
Furthermore, the backup server forms encapsulated data of service data and compressed sub-service data in the encapsulated data through the tree-structured metadata records, wherein the service data is a root node of the tree, and the encapsulated data is a branch node of the tree and compresses the service data into leaf nodes of the tree.
Furthermore, the client requests the backup server to restore all or part of the target service data through the restoration request, the backup server queries target encapsulated data or target compressed sub-service data to be restored according to the restoration request, reads all the compressed sub-service data or the target compressed sub-service data in the target encapsulated data based on the position and the real length of the compressed sub-service data recorded by the compression control block, decompresses all the read compressed sub-service data or the target compressed sub-service data according to the compression method in the compression control block, and feeds back the decompressed data to the corresponding client.
Furthermore, the content of the compression control block is encrypted by an encryption algorithm and then stored.
In a second aspect, the present invention provides a real-time data compression backup apparatus based on the real-time data compression backup method, including: the system comprises at least one compression card, at least two processing units, at least two memories and at least one disk memory, wherein the at least one memory and the at least one processing unit are connected and matched to form a compression data construction module, and the compression data construction module is connected with the compression card and the at least one disk memory.
In a third aspect, the present invention provides a storage medium for implementing a real-time data compression backup method, where the storage medium stores a computer program, and the computer program, when executed by a processor, implements the real-time data compression backup method.
Compared with the prior art, the technical scheme provided by the embodiment of the invention has the following advantages:
the method has the advantages that the business data are compressed in real time after being segmented, when backup is stored, the stored content is the encapsulated data obtained through compression, the data volume of the encapsulated data is small relative to the business data, bandwidth resources are occupied less in the process of writing the encapsulated data into a backup server, time consumption of data writing operation is short, and backup storage efficiency can be effectively improved. The compressed encapsulated data is directly stored, compression after writing the service data into a storage disk and the deletion process of the service data are not needed, and the backup storage execution amount of the backup server is reduced. The compressed encapsulated data is directly stored, so that the storage resources of the backup server can be effectively saved. The compression control block records in the encapsulated data comprise an encrypted compression algorithm, the positions of the compressed sub-service data arranged in the sequence of each compressed sub-service data and the real length of the compressed sub-service data; on one hand, necessary information can be provided for the decompression process, so that the encapsulation data and the recovery of any compressed sub-service data in the encapsulation data are convenient, and on the other hand, the method has certain confidentiality. The capacity of each encapsulated data is the same, so that the storage balance of the backup server is realized conveniently, and the retrieval is facilitated. The backup server forms encapsulation data of service data and compressed sub-service data in the encapsulation data through tree-shaped structured metadata records, wherein the service data is a root node of a tree, the encapsulation data is a branch node of the tree, and the compressed service sub-data is a leaf node of the tree. The encapsulated data of the service data and the corresponding compressed sub-service data in the encapsulated data can be quickly retrieved through the tree-structured metadata.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of a real-time data compression backup method according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating that a compressed data constructing module arranges received compressed sub-service data according to a time sequence and merges the compressed sub-service data into encapsulated data with a fixed capacity according to an embodiment of the present invention;
FIG. 3 is a flowchart of restoring backup data according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating packed data and a compression control block according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a device of a real-time data compression backup method based on the real-time data compression backup method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Example 1
Referring to fig. 1, an embodiment of the present invention provides a real-time data compression backup method, including:
and transmitting the service data to be backed up to the compressed data construction module in a time sequence in a compression request mode. Specifically, a client is configured on the user equipment, and the client supports various operating systems including windows, linux, and macos. The client and the backup server realize communication under the authority limit through a session creating mode so as to support data bidirectional transmission between the client and the backup server.
The client and the backup server establish communication through a compressed session; each compression session binds a compression instance implemented based on a compression data construction module and a compression engine. Each compression instance is configured with a first queue, a second queue, and a third queue.
The client encapsulates the service data into a compression request and sends the compression request to the backup server through a compression session; the first queue of the compression instance accommodates compression request time sequence queuing and time sequence dequeuing; the first queue adopts a FIFO queue, and the compression request which enters the first queue first comes out of the first queue. When the compressed data construction module runs at full load, the compressed data construction module presses the compressed requests which cannot be processed in time into the first queue, and when the compressed data construction module has resources for processing the compressed requests, the first queue provides the compressed requests which firstly enter the first queue to the compressed data construction module.
And the compressed data construction module is used for segmenting the service data to form a plurality of sub-service data arranged in time sequence. Specifically, the compressed data construction module divides the service data received at a set time interval into sub-service data.
The compressed data construction module transmits the sub-service data to the compression engine, the compression engine compresses each sub-service data through a compression algorithm to obtain compressed sub-service data, and the compressed sub-service data is fed back to the compressed data construction module in a time sequence.
In a specific implementation process, the compression engine is implemented based on a compression card. The compression card is hardware specially used for realizing compression coding calculation and is provided with a data input interface and a data output interface. In a specific implementation process, the second queue of the compression example can accommodate time sequence queuing and time sequence dequeuing of the sub-service data, the second queue adopts an FIFO queue, and the sub-service data first enters the first-out second queue of the second queue. And when the compression engine has resources for compressing the sub-service data, the second queue provides the sub-service data which firstly enters the second queue.
And after the compression engine compresses the sub-service data, feeding the compressed sub-service data formed by compressing each sub-service data back to the compressed data construction module. Specifically, the third queue of the compression example may accommodate time sequence queuing and time sequence dequeuing of the compressed sub-service data, and the third queue adopts an FIFO queue, and the compressed sub-service data that enters the third queue first comes first into the third queue first comes first out. And when the compressed data construction module has resources for processing the compressed sub-service data, the third queue provides the compressed sub-service data which enters the third queue firstly.
And the compressed data construction module arranges the received compressed sub-service data according to a time sequence and combines the compressed sub-service data into encapsulated data with fixed capacity. Wherein encapsulating the data comprises: compressing sub-service data, compressing control blocks and filling zeros which are arranged in time sequence; the padding zero is used for padding the encapsulation data to a preset fixed capacity, and the record of the compression control block comprises a compression algorithm, the position of the compressed sub-service data and the real length of the compressed sub-service data. The position of the compressed sub-service data is identified by the offset of the first bit, and the position of each compressed sub-service data and the real length of the compressed sub-service data are arranged according to the time sequence of the compressed sub-service data.
In a specific implementation process, referring to fig. 2, the compressed data construction module counts the total amount of the compressed sub-service data currently sorted; the compressed data construction module receives the next time sequence compressed sub-service data to obtain the volume of the next time sequence compressed sub-service data; if the quantity of the compressed sub-service data of the next time sequence is accumulated until the total quantity of the compressed sub-service data of the current sequence is larger than the fixed capacity of the encapsulated data, counting the information of the compressed sub-service data of the current sequence to obtain the content of a compressed control block, encrypting the content of the compressed control block and encapsulating the encrypted content to obtain the compressed control block, and subtracting the total quantity of the compressed sub-service data of the current sequence and the quantity of the compressed control block from the fixed capacity to obtain the quantity of the filled zeros; and the compressed data construction module encapsulates the currently sequenced compressed sub-service data, the padding zeros and the corresponding compression control block to obtain encapsulated data. Specifically, referring to fig. 4, the compression control block is encapsulated at the tail of the encapsulated data, and the padding zeros are located between the compressed sub-service data and the compression control block. And if the amount of the compressed sub-service data of the next time sequence is accumulated to the total amount of the compressed sub-service data of the current sequence is smaller than the fixed capacity of the encapsulated data, adding the compressed sub-service data of the next time sequence to the tail part of the compressed sub-service data of the current sequence.
The implementation of the compression example of the application comprises a process of segmenting the service data according to a time sequence to obtain sub-service data, a process of compressing the sub-service data according to the time sequence to obtain compressed sub-service data, a process of feeding the compressed sub-service data back to the compressed data construction module according to the time sequence, and a process of arranging the compressed sub-service data according to the time sequence and packaging the compressed sub-service data into packaged data with a fixed capacity by the compressed data construction module.
And the compressed data construction module writes the encapsulated data into a storage module of the backup server. The storage modules of the backup server include, but are not limited to, Raid-created storage pools, disk-created virtual tape libraries, and disks.
The backup server constructs tree-shaped structured metadata according to the content of the compression control block, records information forming service data, information forming encapsulated data of the service data and information of compressed sub-service data in the encapsulated data through the metadata, wherein the service data is a root node of the tree, the encapsulated data is a branch node of the tree, and the compressed sub-service data is a leaf node of the tree. And the backup data is convenient to search and position.
In a specific implementation process, referring to fig. 3, a process of restoring backup data is as follows:
and the client requests the backup server to restore all or part of the target service data through the restoration request. The content of the restoration request may be information of the target service data, and when the content of the restoration request is information of the target service data, all encapsulated data related to the target service data is restored, and the content of the restoration request may be information of data to be restored in the target service data.
And the backup server analyzes the reduction request to obtain target encapsulation data or target compression sub-service data to be reduced.
And quickly inquiring target encapsulation data or target compression sub-service data to be restored according to the tree-structured metadata.
And the backup server reads all the compressed sub-service data or the target compressed sub-service data in the target encapsulated data based on the position and the real length of the compressed sub-service data recorded by the compression control block.
Decompressing all the read compressed sub-service data or the target compressed sub-service data according to the compression method in the compression control block, and feeding back to the corresponding client.
Example 2
Referring to fig. 5, an embodiment of the present invention provides a real-time data compression backup apparatus based on the real-time data compression backup method, including: the interface unit, the at least one compression card, the at least two processing units, the at least two memories and the at least one disk memory are connected through a bus. Wherein,
the system is connected with the client through the interface unit, establishes a session after the authority of the client is verified, and realizes bidirectional data transmission with the client based on the session.
The at least one memory and the at least one processing unit are connected and matched to form a compressed data construction module, and the compressed data construction module is used for segmenting the service data to form a plurality of sub-service data arranged in time sequence. The compressed data construction module is connected with the compression card and at least one disk memory. The compressed data construction module transmits the sub-service data to the compression card, the compression engine based on the compression card compresses each sub-service data through a compression algorithm to obtain compressed sub-service data, and the compressed sub-service data is fed back to the compressed data construction module in a time sequence. And the compressed data construction module arranges the received compressed sub-service data according to a time sequence and combines the compressed sub-service data into encapsulated data with fixed capacity. The compressed data construction module writes the encapsulated data to disk storage.
Example 3
The embodiment of the invention provides a storage medium for realizing a real-time data compression backup method, wherein the storage medium stores a computer program, and the computer program realizes the real-time data compression backup method when being executed by a processor.
The business data is compressed in real time after being segmented, when backup is stored, the stored content is encapsulated data obtained through compression, the data volume of the encapsulated data is small relative to the business data, bandwidth resources are occupied less in the process of writing the encapsulated data into a backup server, the time consumption of data writing operation is short, and the backup storage efficiency can be effectively improved. The compressed encapsulated data is directly stored, the compression after the service data is written into a storage disk and the deletion process of the service data are not needed, and the backup storage execution amount of the backup server is reduced. The compressed encapsulated data is directly stored, so that the storage resources of the backup server can be effectively saved. The compression control block records in the encapsulated data comprise an encrypted compression algorithm, the positions of the compressed sub-service data arranged in the sequence of each compressed sub-service data and the real length of the compressed sub-service data; on one hand, necessary information can be provided for the decompression process, so that the encapsulation data and the recovery of any compressed sub-service data in the encapsulation data are convenient, and on the other hand, the method has certain confidentiality. The encapsulated data books have the same capacity, so that the storage balance of the backup server is realized conveniently, and the retrieval is facilitated. The backup server forms encapsulated data of service data and compressed sub-service data in the encapsulated data through tree-shaped structured metadata records, wherein the service data is a root node of a tree, and the encapsulated data is a branch node of the tree and compresses the service sub-data into leaf nodes of the tree. The encapsulated data of the service data and the corresponding compressed sub-service data in the encapsulated data can be quickly retrieved through the tree-structured metadata.
In the embodiments provided herein, it should be understood that the disclosed structures and methods may be implemented in other ways. For example, the above-described structural embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, structures or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A real-time data compression backup method is characterized by comprising the following steps: the method comprises the steps that service data to be backed up are transmitted to a compressed data construction module in a time sequence mode in a compression request mode, and the compressed data construction module is used for segmenting the service data to form a plurality of sub-service data arranged in a time sequence mode;
the compressed data construction module transmits the sub-service data to a compression engine, the compression engine compresses each sub-service data through a compression algorithm to obtain compressed sub-service data, and the compressed sub-service data is fed back to the compressed data construction module in a time sequence;
the compressed data construction module arranges the received compressed sub-service data according to time sequence and combines the compressed sub-service data into encapsulated data with fixed capacity; wherein encapsulating the data comprises: compressing sub-service data, compressing control blocks and filling zeros which are arranged in time sequence; the zero padding is used for padding the encapsulated data to a fixed capacity, and the record content of the compression control block comprises a compression algorithm, the positions of the compressed sub-service data arranged according to the sequence of each compressed sub-service data and the real length of the compressed sub-service data;
and the compressed data construction module writes the encapsulated data into a disk memory of the backup server.
2. The real-time data compression backup method according to claim 1, characterized in that communication is established between the client and the backup server through a compression session, the client encapsulates the service data into a compression request, and the compression request is sent to the backup server through the compression session;
the backup server will bind the compression session based on the compression instance implemented by the compression data construction module and the compression engine.
3. The real-time data compression backup method according to claim 2, wherein each compression instance is configured with a first queue for queuing compression requests, the first queue providing compression requests in order for the compressed data construction module; each compression instance is provided with a second queue for queuing the sub-service data, and the second queue provides the sub-service data for the compression engine in order; each compression example is provided with a third queue for queuing the compressed sub-service data, and the third queue provides the compressed sub-service data for the compressed data construction module in order; the first queue, the second queue and the third queue are all FIFO queues.
4. The real-time data compression backup method according to claim 1, wherein the compressed data construction module counts the total amount of the currently ordered compressed sub-service data, when the amount of the next time sequence of compressed sub-service data is added to the total amount of the currently ordered compressed sub-service data and is greater than the fixed capacity of the encapsulated data, counts the information of the currently ordered compressed sub-service data to obtain the content of the compressed control block, constructs the compressed control block, the fixed capacity subtracts the total amount of the currently ordered compressed sub-service data and the amount of the compressed control block to obtain the amount of zero padding, and the compressed data construction module encapsulates the currently ordered compressed sub-service data, fills zeros, and the corresponding compressed control block to obtain the encapsulated data.
5. The method for real-time data compression backup according to claim 4, wherein the compression control block is encapsulated at the tail of the encapsulated data, and the padding zeros are located between the compressed sub-service data and the compression control block.
6. The real-time data compression backup method according to claim 1, wherein tree-structured metadata is constructed according to the content of the compression control block, and information constituting service data, information constituting encapsulation data of the service data, and information compressing sub-service data in the encapsulation data are recorded by the metadata, wherein the service data is a root node of the tree, the encapsulation data is a branch node of the tree, and the encapsulation data compresses the service sub-data into a leaf node of the tree.
7. The real-time data compression backup method according to claim 1, wherein the client requests the backup server to restore all or part of the target service data through the restoration request, the backup server queries the target encapsulated data or the target compressed sub-service data to be restored according to the restoration request, reads all the compressed sub-service data or the target compressed sub-service data in the target encapsulated data based on the position and the real length of the compressed sub-service data recorded by the compressed control block, decompresses all the read compressed sub-service data or the target compressed sub-service data according to the compression method in the compressed control block, and feeds back the decompressed data to the corresponding client.
8. The real-time data compression backup method as claimed in claim 1, wherein the recording content of the compression control block is encrypted by an encryption algorithm and stored.
9. A real-time data compression backup device based on any one of the real-time data compression backup methods of claims 1 to 8, comprising: the system comprises at least one compression card, at least two processing units, at least two memories and at least one disk memory, wherein the at least one memory and the at least one processing unit are connected and matched to form a compression data construction module, and the compression data construction module is connected with the compression card and the at least one disk memory.
10. A storage medium for implementing a method for compressed backup of real-time data, the storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a method for compressed backup of real-time data according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210762078.1A CN114968669B (en) | 2022-06-30 | 2022-06-30 | Real-time data compression backup method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210762078.1A CN114968669B (en) | 2022-06-30 | 2022-06-30 | Real-time data compression backup method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114968669A true CN114968669A (en) | 2022-08-30 |
| CN114968669B CN114968669B (en) | 2024-10-22 |
Family
ID=82967534
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210762078.1A Active CN114968669B (en) | 2022-06-30 | 2022-06-30 | Real-time data compression backup method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114968669B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109062502A (en) * | 2018-07-10 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of data compression method, device, equipment and computer readable storage medium |
| US10581751B1 (en) * | 2015-12-16 | 2020-03-03 | EMC IP Holding Company LLC | Multi-queue based system for throttling backup agent save processes |
| CN114449052A (en) * | 2022-01-28 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Data compression method and device, electronic equipment and storage medium |
-
2022
- 2022-06-30 CN CN202210762078.1A patent/CN114968669B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10581751B1 (en) * | 2015-12-16 | 2020-03-03 | EMC IP Holding Company LLC | Multi-queue based system for throttling backup agent save processes |
| CN109062502A (en) * | 2018-07-10 | 2018-12-21 | 郑州云海信息技术有限公司 | A kind of data compression method, device, equipment and computer readable storage medium |
| CN114449052A (en) * | 2022-01-28 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Data compression method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114968669B (en) | 2024-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8473652B2 (en) | Systems and methods for compression of data for block mode access storage | |
| US7970965B2 (en) | Method and system for compression of data for block mode access storage | |
| US8380900B2 (en) | Systems and methods for compressing files for storage and operation on compressed files | |
| CA2564844C (en) | Method and system for compression of files for storage and operation on compressed files | |
| JP6106901B2 (en) | Data processing method and device in distributed file storage system | |
| US10268398B2 (en) | Storage system, recording medium for storing control program and control method for storage system | |
| US8250043B2 (en) | System and method for compression of partially ordered data sets | |
| US20060190643A1 (en) | Method and system for compression of data for block mode access storage | |
| US11442627B2 (en) | Data compression utilizing low-ratio compression and delayed high-ratio compression | |
| EP1949541A2 (en) | Method and system for compression of logical data objects for storage | |
| US7657533B2 (en) | Data management systems, data management system storage devices, articles of manufacture, and data management methods | |
| US20210011637A1 (en) | Selective compression and encryption for data replication | |
| CN113535068A (en) | Data reading method and system | |
| CN114610679A (en) | Storage device, data storage method thereof and cloud storage system | |
| CN114968669B (en) | Real-time data compression backup method, device and storage medium | |
| WO2024169851A1 (en) | Data compression method, system, and device, and computer readable storage medium | |
| CN118819418A (en) | File reading method, device, computer equipment and readable storage medium | |
| CN116257180A (en) | Data access method and device | |
| JPH0520146A (en) | Electronic bulletin board device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |