CN113778331A - Data processing method, main node and storage medium - Google Patents
Data processing method, main node and storage medium Download PDFInfo
- Publication number
- CN113778331A CN113778331A CN202110925774.5A CN202110925774A CN113778331A CN 113778331 A CN113778331 A CN 113778331A CN 202110925774 A CN202110925774 A CN 202110925774A CN 113778331 A CN113778331 A CN 113778331A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- written
- slave
- master node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 29
- 238000012790 confirmation Methods 0.000 claims abstract description 51
- 238000000034 method Methods 0.000 claims description 43
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 5
- 238000010200 validation analysis Methods 0.000 claims description 3
- 238000005192 partition Methods 0.000 description 22
- 238000011084 recovery Methods 0.000 description 10
- 230000002085 persistent effect Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data processing method, which comprises the following steps: a main node in a cluster receives and writes data to be written sent by a client; the master node sends the data to be written to at least one slave node in the cluster; if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written; by the data processing method, the main node and the storage medium, the data writing times can be reduced when data are written, and further delay caused by multiple writing operations is reduced.
Description
Technical Field
The present application relates to the field of distributed storage technologies, and in particular, to a data processing method, a host node, and a storage medium.
Background
In a conventional log-based data model, the process of writing data into a disk requires two steps, namely, firstly, writing the data and operation information into a log partition as a log, and then sending the log to other slave nodes of a cluster copy group by a master node in the cluster copy group. Other slave nodes will also write this log into the log partition and send response information to the master node. After the second member in the copy group confirms that the write operation can be executed, the main node sends a write instruction to other slave nodes, and the other slave nodes move the data from the log partition to the data partition.
The model has the advantages that the strong consistency of the data can be ensured, and the incremental recovery of the data can be carried out based on the log when the system fails; however, in a scenario where one log writes data based on the above model, there is a problem that the number of write operations is large, which increases write data delay.
Disclosure of Invention
The present application provides a data processing method, a host node, and a storage medium, so as to at least solve the above technical problems in the prior art.
A first aspect of the present application provides a data processing method, including:
the master node sends the data to be written to at least one slave node in the cluster;
if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written.
In the above scheme, the master node and/or the first slave node writes the version identifier corresponding to the data to be written.
In the foregoing solution, after the master node receives first acknowledgement information sent by a first slave node in all the slave nodes, the method further includes:
the master node confirms that a data block corresponding to the data to be written is valid in the first slave node;
and/or the master node confirms that a data block corresponding to the data to be written fails at a second slave node, wherein the second slave node is a slave node except the first slave node in all the slave nodes.
In the above scheme, the method further comprises:
the master node acquires a data block corresponding to each slave node;
and the master node determines the validity of each data block according to the attribute information of the data block corresponding to each slave node to obtain a confirmation result.
In the foregoing solution, if the confirmation result indicates that at least one of the data blocks is invalid, the method further includes:
the master node confirms the version identifications corresponding to the data blocks in all the nodes included in the cluster respectively;
determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time;
if the first node is the master node, the master node determines a third slave node, and the version identifier of the data block corresponding to the third slave node is not the version identifier with the latest update time;
and sending the data corresponding to the version identification with the latest updating time to the third slave node.
In the foregoing solution, if the confirmation result indicates that at least one of the data blocks is invalid, the method further includes:
the master node confirms the version identifications corresponding to the data blocks in all the nodes included in the cluster respectively;
determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time;
if the first node is not the master node, the master node writes a data block corresponding to the version identifier with the latest updating time;
the master node determines a third slave node, wherein the version identification of the data block corresponding to the third slave node is not the version identification with the latest updating time;
and sending the data corresponding to the version identification with the latest updating time to the third slave node.
In the foregoing solution, after the data to be written is confirmed to be successfully written into the cluster, the method further includes:
the main node sends second confirmation information to the client; the second acknowledgement information is used to indicate that the cluster successfully writes the data to be written.
A second aspect of the present application provides a master node, comprising:
the receiving unit is used for receiving and writing data to be written sent by the client;
a sending unit, configured to send the data to be written to at least one slave node in the cluster;
the confirming unit is used for confirming that the data to be written is successfully written into the cluster if first confirming information sent by a first slave node in all the slave nodes is received; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written.
A third aspect of the present application provides an electronic device comprising: a memory for storing executable instructions; and the processor is used for realizing the method steps of the data processing method when executing the executable instructions stored in the memory.
A fourth aspect of the present application provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the method steps of the data processing method as described above.
By the data processing method, the main node in the cluster receives and writes data to be written sent by the client; the master node sends the data to be written to at least one slave node in the cluster; if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written. In this way, when data is written, the number of data writing operations can be reduced, and the write data delay caused by multiple writing operations can be reduced.
Drawings
FIG. 1 is a flow chart illustrating a related art process of writing data to a disk;
FIG. 2 is a first alternative flow chart of a data processing method provided by the embodiment of the present application;
FIG. 3 is a second alternative flow chart of the data processing method according to the embodiment of the present application;
FIG. 4 is a third alternative flow chart of the data processing method according to the embodiment of the present application;
fig. 5 is a fourth alternative flowchart of a data processing method according to an embodiment of the present application;
FIG. 6 is a fifth alternative flowchart of a data processing method according to an embodiment of the present disclosure;
fig. 7 shows a sixth alternative flowchart of a data processing method provided in an embodiment of the present application;
fig. 8 is a schematic diagram illustrating an alternative structure of a master node according to an embodiment of the present disclosure;
fig. 9 is a schematic diagram illustrating a hardware component structure of a master node according to an embodiment of the present application.
Detailed Description
In order to make the objects, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the conventional log-based data model, when a user modifies data (including writing new data and modifying written data), data and operation information are written into a local log partition of each node in a cluster as a log, and then the data is written into a corresponding data partition after a copy group confirms that the operation (write operation) can be performed. The above operation is generally called a Write Ahead Log (WAL), and the WAL operation causes a problem of Write amplification.
The process of writing data into a disk needs two steps, firstly, data and operation information are written into a log partition as a log, and then a master node in a cluster copy group sends the log to other slave nodes of the cluster copy group. Other slave nodes will also write this log into the log partition and send response information to the master node. After the second member in the copy group confirms that the write operation can be executed, the main node sends a write instruction to other slave nodes, and the other slave nodes move the data from the log partition to the data partition.
Specifically, in order to ensure consistency of multiple copies of data, a technical solution that is usually adopted is based on log implementation. Before writing Data into the local persistent storage media of each Node, the system needs to write the Operation (Operation), Version identifier (Version), real Data (Data) and the like of a user into the Log area of the master Node (Primary Node) in the form of Log (Log), and then the master Node sends the Log to each slave Node (Replicated Node).
If more than half of the slave nodes receive the master node's log and respond, the master node can confirm that the write operation can be performed correctly accordingly. Thereafter, the master node writes the user data to the data partition and sends a message confirming the writing to each slave node. Fig. 1 is a schematic flow chart showing a related art process of writing data to a magnetic disk, which will be described according to various steps.
Step S101, the client sends data to be written to the main node.
The master node writes the data to be written and the operation information into a Log partition (Log partition) in a Log mode, wherein the operation information at least comprises one of the time of writing the data, the data source and at least one slave node writing the data.
And step S102, the master node sends the logs to all the slave nodes.
Step S103, each slave node writes the log.
And each slave node writes the log into the log partition of the slave node, and sends confirmation information to the master node after confirming that the log can be written.
In step S104, the master node confirms the write information.
And if the master node receives the confirmation information sent by more than half slave nodes, confirming that the data to be written can be written into the slave nodes, and sending a data writing request to each slave node.
In step S105, data is written.
After receiving the write-in data request, each slave node transfers the data to be written from the log partition to the data partition to complete data write-in operation; and/or the main node transfers the data to be written from the log partition to the data partition to complete the data writing operation.
Through the above process, it can be seen that, for one data write operation, the data write operation is disassembled into two disk write operations at the bottom layer, one is written into the log area (steps S101 to S103), and the second is written into the data partition (step S105). Although the requirement of strong consistency is ensured, the problem of write amplification is brought, one part of data is written twice in the bottom layer, the write performance of the system is reduced, and the write delay is increased.
In some scenarios sensitive to the write latency, the solutions in the related art cannot meet the requirements of the user.
Therefore, aiming at the defects existing in the current distributed storage technology, the application provides a data processing method which can overcome part or all of the defects in the prior art.
Fig. 2 shows a first alternative flowchart of a data processing method provided in an embodiment of the present application, which will be described according to various steps.
Step S201, the master node in the cluster receives and writes the data to be written sent by the client.
In some embodiments, the cluster comprises one master node and at least one slave node; when data to be written exists, data transmission is generally performed between a client and a master node, and the client transmits the data to be written to the master node.
In some optional embodiments, the master node may further write the data to be written into a persistent storage medium corresponding to the master node; the master node can also write the version identification corresponding to the data to be written into a persistent storage medium corresponding to the master node.
The version identifier corresponding to the data to be written can be used for representing the number of times of writing the data to be written, and the version identifier corresponding to the data to be written can be monotonically increased along with the number of times of writing operation; the version identification corresponding to the to-be-written data can be used for comparing versions during data recovery so as to confirm the data corresponding to the latest version identification; the version identifier corresponding to the to-be-written data may also be used to confirm a node that misses data corresponding to the latest version identifier.
Step S202, the master node sends the data to be written to at least one slave node in the cluster.
In some embodiments, the master node sends the data to be written and/or a version identifier corresponding to the data to be written to at least one slave node in the cluster; the master node may send the data to be written to all slave nodes in the cluster, or may send the data to be written to some slave nodes in the cluster.
Step S203, if the master node receives first confirmation information sent by a first slave node of all the slave nodes, it confirms that the data to be written is successfully written into the cluster.
In some embodiments, after any slave node receives the data to be written, the data to be written is written into the slave node; if any slave node also receives the version identifier corresponding to the data to be written, which is sent by the master node, any slave node can also write the version identifier corresponding to the data to be written into the slave node. The writing of the data to be written and/or the version identification into the slave node means that the data to be written and/or the version identification are written into a data partition of the slave node. And the slave node sends first confirmation information to the master node under the condition that the data to be written is successfully written.
In specific implementation, if the master node receives first confirmation information sent by a first slave node in all the slave nodes (or multiple slave nodes allowing to write the data to be written), it is confirmed that the data to be written is successfully written into the cluster; the first slave node may be any slave node in the cluster that successfully writes data to be written. Optionally, after the first slave node writes the data to be written in, a data block (Chunk) corresponding to the data to be written is generated, and after the master node receives first acknowledgement information sent by the first slave node, the data block corresponding to the data to be written in the first slave node is marked as valid.
And if the master node does not receive the first confirmation information sent by any slave node, confirming that the data to be written is not successfully written into the cluster, and marking the data blocks corresponding to the data to be written in all the slave nodes as invalid.
If the master node does not receive confirmation information sent by a second slave node, confirming that a data block corresponding to the data to be written fails at the second slave node; the second slave node is a slave node other than the first slave node among all the slave nodes.
The first confirmation information is used for indicating that the first slave node successfully writes the data to be written; the validity (valid or invalid) of the data block corresponding to the data to be written is used for subsequent data recovery. The first slave node may include one slave node, and may also include at least one slave node.
In some embodiments, if the master node confirms that the data to be written is successfully written into the cluster, the master node may further send second confirmation information to the client; the second acknowledgement information is used to indicate that the cluster successfully writes the data to be written.
Therefore, by the data processing method provided by the embodiment of the application, the main node in the cluster receives and writes the data to be written sent by the client; the master node sends the data to be written to at least one slave node in the cluster; if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written. When data is written, extra performance overhead caused by writing the data into the log partition is avoided, the performance of the disk can be utilized to the maximum extent, the writing efficiency is improved, and the method is suitable for scenes with high requirements on writing delay.
In some embodiments, for a slave node that does not successfully write the data to be written, the embodiments of the present application further provide a data processing method, which can recover the data to be written at each node.
Fig. 3 shows a second alternative flowchart of the data processing method provided in the embodiment of the present application, which will be described according to various steps.
Step S301, the master node acquires a data block corresponding to each slave node.
In some embodiments, the master node acquires a data block corresponding to each slave node; and/or attribute information of the data block corresponding to each slave node.
Wherein the attribute information of the data block comprises that the data block is valid or the data block is invalid. Alternatively, the attribute information of the data block may be set by an extended attribute of an underlying file system (such as ext4, xfs, etc.). Specifically, the master node may mark the validity of the data block by marking a value of a key (e.g., isValid) of an extended attribute of the data block. For example, a key value of 1 indicates that the data block is valid; a key value of 0 indicates that the data block is invalid (or stale).
Step S302, the main node determines the validity of each data block according to the attribute information of the data block corresponding to each slave node, and a confirmation result is obtained.
In some embodiments, the master node confirms the validity of each data block according to the attribute information of the data block corresponding to each slave node, and obtains a confirmation result.
In some optional embodiments, after confirming the failed data block, the master node may further add a node corresponding to the failed data block to the queue to be recovered.
In some embodiments, if the validation result indicates that at least one of the data blocks is invalid, the method may further include:
step S303, confirming the version identifiers respectively corresponding to the data blocks in all the nodes included in the cluster.
In some embodiments, all the nodes include a master node and all the slave nodes, and the master node confirms the version identifiers respectively corresponding to the data blocks in all the nodes included in the cluster.
In some embodiments, when the node writes the data to be written, the node also writes the version identifier corresponding to the data to be written. Whether the data to be written is written or not, the nodes generate corresponding data blocks; the difference is that corresponding data blocks are the same in the nodes in which the data to be written is successfully written; and the data block fails in the node which is not successfully written with the data to be written.
In some embodiments, version identifiers of data blocks corresponding to the same data to be written may be different at different nodes. For example, for the data to be written in the first version identifier, the slave node a writes successfully, and the slave node B does not write successfully, at this time, the data block corresponding to the slave node a is valid, and the data block corresponding to the slave node B is invalid; for the data to be written in the second version identifier, the slave node a is not successfully written in, the slave node B is successfully written in, and at this time, the data blocks corresponding to the slave node a and the slave node B are both valid, except that the version identifier corresponding to the slave node a is the first version identifier, and the version identifier corresponding to the slave node B is the second version identifier.
Step S304, determining the first node corresponding to the data block corresponding to the version identifier with the latest updating time.
In some embodiments, the master node determines that the most recent version in time of update identifies the first node to which the corresponding data block corresponds.
Taking the version identifier as a number as an example, if the version identifier is monotonically increased, the master node may determine the data block with the largest version identifier as the data block with the latest update time.
In step S305, it is determined whether the first node is a master node.
In some embodiments, the master node determines whether the first node is a master node; if the first node is a master node, executing step S306; if the first node is not the master node, step S307 is executed.
In step S306, the master node determines a third slave node.
In some embodiments, if the first node is the master node, the master node determines a third slave node, where the version identification of the data block corresponding to the third slave node is not the version identification with the latest update time; and sending the data corresponding to the version identification with the latest updating time to the third slave node.
In some embodiments, after the slave nodes write data successfully, the slave nodes may further send confirmation information to the master node, and the master node marks, according to the confirmation information, the validity of the data block corresponding to the version identifier whose update time is the latest in each slave node.
Step S307, the master node writes data corresponding to the version identifier with the latest update time.
In some embodiments, the master node writes data corresponding to the latest version identifier and/or the latest version identifier.
In some embodiments, the master node may further determine a third slave node that does not include the version identification corresponding to the most recent update time; and sending the data corresponding to the version identification with the latest updating time to the third slave node. Wherein the version identification of the data block corresponding to the third slave node is not the version identification with the latest update time.
In some embodiments, after the slave nodes write data successfully, the slave nodes may further send confirmation information to the master node, and the master node marks, according to the confirmation information, the validity of the data block corresponding to the version identifier whose update time is the latest in each slave node.
In some optional embodiments, steps S301 to S307 may be set to be executed within a first time threshold range (e.g., morning or idle time), so as to avoid preempting disk bandwidth and central processor resources. Optionally, the execution time and/or frequency of steps S301 to S307 may be set based on the priority of the data to be written, so as to reduce the influence of steps S301 to S307 on the normal service of the client.
For example, for data to be written with high priority can be executed at a time outside the first time threshold range, and the execution frequency is higher than that of the data to be written with low priority, so that the problem that a program corresponding to the data with high priority cannot be normally executed when the data writing is inconsistent due to a cluster failure is avoided.
Therefore, by the data processing method provided by the embodiment of the application, whether data needs to be recovered or not is confirmed through silent scanning, nodes which need to recover data are further determined, and data recovery is performed. When data writing is inconsistent due to the fact that a cluster fails and the like, final consistency of the data in the cluster can be guaranteed through full recovery of the data and a version ratio peer-to-peer mode.
Fig. 4 shows a third alternative flowchart of the data processing method provided in the embodiment of the present application, which will be described according to various steps.
Step S401, the client sends the data to be written to the main node in the cluster.
In some optional embodiments, the master node may further write the data to be written into a persistent storage medium corresponding to the master node, and/or the master node may further write a version identifier corresponding to the data to be written into the persistent storage medium corresponding to the master node.
The version identifier corresponding to the data to be written can be used for representing the number of times of writing the data to be written, and the version identifier corresponding to the data to be written can be monotonically increased along with the number of times of writing operation; the version identification corresponding to the to-be-written data can be used for comparing versions during data recovery so as to confirm the data corresponding to the latest version identification; the version identifier corresponding to the to-be-written data may also be used to confirm a node that misses data corresponding to the latest version identifier.
Step S402, the master node sends the data to be written to at least one slave node in the cluster.
In some embodiments, the master node sends the data to be written and/or a version identifier corresponding to the data to be written to at least one slave node in the cluster; the master node may send the data to be written to all slave nodes in the cluster, or may send the data to be written to some slave nodes in the cluster.
In step S403, the slave node transmits first confirmation information to the master node.
In some embodiments, after any slave node receives the data to be written, the data to be written is written into the slave node; if any slave node also receives the version identifier corresponding to the data to be written, which is sent by the master node, any slave node can also write the version identifier corresponding to the data to be written into the slave node. The writing of the data to be written and/or the version identification into the slave node means that the data to be written and/or the version identification are written into a data partition of the slave node. And the slave node sends first confirmation information to the master node under the condition that the data to be written is successfully written.
Therefore, by the data processing method provided by the embodiment of the application, the main node in the cluster receives and writes the data to be written sent by the client; the master node sends the data to be written to at least one slave node in the cluster; if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written. When data is written, extra performance overhead caused by a write log is avoided, the performance of a magnetic disk can be utilized to the maximum extent, the data writing efficiency is improved, and the method is suitable for scenes with high requirements on data writing delay.
Fig. 5 shows a fourth alternative flowchart of the data processing method provided in the embodiment of the present application, which will be described according to various steps.
Step S501, the client sends the data to be written to the main node in the cluster.
Step S502, the main node writes the data to be written.
In some optional embodiments, the master node may further write the data to be written into a persistent storage medium corresponding to the master node, and the master node may further write a version identifier corresponding to the data to be written into the persistent storage medium corresponding to the master node.
The version identifier corresponding to the data to be written can be used for representing the number of times of writing the data to be written, and the version identifier corresponding to the data to be written can be monotonically increased along with the number of times of writing operation; the version identification corresponding to the to-be-written data can be used for comparing versions during data recovery so as to confirm the data corresponding to the latest version identification; the version identifier corresponding to the to-be-written data may also be used to confirm a node that misses data corresponding to the latest version identifier.
In some optional embodiments, if the master node fails to write the data to be written, the master node may further send a retransmission request to the client to request the client to send the data to be written to the master node again.
In step S503, the master node transmits data to be written to at least one slave node.
Step S504, the slave node writes the data to be written.
In some embodiments, after receiving the data to be written and the version identifier corresponding to the data to be written, the at least one slave node writes the data to be written and the version identifier corresponding to the data to be written into the slave node.
And step S505, when the slave node successfully writes, the slave node sends first confirmation information to the master node.
Step S506, the master node sends second confirmation information to the client.
In some embodiments, if the master node confirms that the data to be written is successfully written into the cluster, the master node may further send second confirmation information to the client; the second acknowledgement information is used to indicate that the cluster successfully writes the data to be written.
In specific implementation, if the master node receives first confirmation information sent by a first slave node in all the slave nodes (or multiple slave nodes allowing to write the data to be written), it is confirmed that the data to be written is successfully written into the cluster; optionally, after the data to be written is written in by the first slave node, a corresponding data block is generated, and after the master node receives first acknowledgement information sent by the first slave node, the data block corresponding to the data to be written in the first slave node is marked as valid.
Or if the master node does not receive the first confirmation information sent by any slave node, confirming that the data to be written is not successfully written into the cluster, and marking the data blocks corresponding to the data to be written in all the slave nodes as invalid.
In a specific implementation, the master node may mark the validity of the data block by marking a value of a key (e.g., isValid) of an extended attribute of the data block. For example, a value of 1 indicates that the data block is valid; a value of 0 indicates a data block failure (or invalidation). Optionally, the master node may be set by an extended attribute of an underlying file system (e.g., ext4, xfs, etc.), specifically, a key of the extended attribute is isValid, a value of 1 indicates that Chunk is valid, and a value of 0 indicates that Chunk is invalid.
The first confirmation information is used for indicating that the first slave node successfully writes the data to be written; the validity (valid or invalid) of the data block corresponding to the data to be written is used for subsequent data recovery. The first slave node may include one slave node, or may include two or more slave nodes.
In some embodiments, in order to timely count the Chunk with the failed storage system, the embodiment of the present application may further set a timing task, configured to execute Chunk silent scanning (scrub), scan the Chunk corresponding to each node according to the timing task, read the extended attribute of the Chunk, determine the validity of the Chunk, and determine that the Chunk is a valid Chunk or a failed Chunk. In other embodiments, when the master node performs a read data operation, the extended attribute of Chunk may also be read to determine the validity of Chunk.
Fig. 6 shows a fifth alternative flowchart of the data processing method provided in the embodiment of the present application, which will be described according to various steps.
In step S601, the master node acquires a data block list.
In some embodiments, the data block list includes information of data blocks corresponding to each slave node corresponding to a master node in the cluster. The information of the data block may include a node identifier of a node corresponding to the data block, and may further include attribute information of the data block, for example, the data block is valid or invalid.
In step S602, the master node reads the value of each data block.
In some embodiments, the master node reads the "isValid" value in the metadata corresponding to each data block and validates the validity of each data block based on the "isValid" value.
In other embodiments, the master node obtains a data block corresponding to each slave node and attribute information of the data block corresponding to each slave node.
Wherein the attribute information of the data block comprises that the data block is valid or the data block is invalid.
In some optional embodiments, the master node further determines whether values of all data blocks have been read, and if all values have been read, the process is ended; if not, step S602 is repeated.
Step S603, add the node corresponding to the failed data block to the queue to be recovered.
In some embodiments, the master node adds the node corresponding to the failed data block to the queue to be restored after validating the failed data block.
In some embodiments, if a data block is determined to be invalid, a data recovery operation needs to be performed. Fig. 7 shows a sixth alternative flowchart of the data processing method provided in the embodiment of the present application, which will be described according to various steps.
Step S801, the master node confirms the version identifiers respectively corresponding to the data blocks in all the nodes included in the cluster.
In some embodiments, the master node confirms the version identifiers respectively corresponding to the data blocks in all the nodes included in the cluster; wherein all nodes comprise the master node and at least one slave node corresponding to the master node.
In some embodiments, when the node writes the data to be written, the node also writes the version identifier corresponding to the data to be written. Whether the data to be written is written or not, the nodes generate corresponding data blocks; the difference is that in the node in which the data to be written is successfully written, the data block corresponding to the data to be written is valid; the data blocks in the nodes in which the data to be written are successfully written are the same; and in the nodes which are not successfully written with the data to be written, the data blocks corresponding to the data to be written are invalid.
In some embodiments, version identifiers of data blocks corresponding to the same data to be written may be different at different nodes. For example, for the data to be written in the first version identifier, the slave node a writes successfully, and the slave node B does not write successfully, at this time, the data block corresponding to the slave node a is valid, and the data block corresponding to the slave node B is invalid; for the data to be written in the second version identifier, the slave node a is not successfully written in, the slave node B is successfully written in, and at this time, the data blocks corresponding to the slave node a and the slave node B are both valid, except that the version identifier corresponding to the slave node a is the first version identifier, and the version identifier corresponding to the slave node B is the second version identifier.
Step S802, determine the data block corresponding to the version identifier with the latest update time.
In some embodiments, taking the version identifier as a number as an example, if the version identifier monotonically increases with the number of writes, the master node may confirm the data block with the largest version identifier as the data block with the latest update time.
In some optional embodiments, the master node determines that the version with the latest update time identifies the first node to which the corresponding data block corresponds.
In step S803, it is confirmed whether the first node is a master node.
In some embodiments, the master node determines whether the first node is a master node; if the first node is the master node, performing step S806; if the first node is not the master node, step S804 is executed.
In step S804, the master node confirms the first node.
In some embodiments, the master node determines, based on the version identification, a first node corresponding to a data block with a latest update time.
In step S805, the master node writes data corresponding to the version identifier whose update time is the latest.
In some embodiments, the master node writes data corresponding to the version identifier with the latest update time, and writes the version identifier with the latest update time.
In step S806, the master node determines a third slave node, where the version id of the data block corresponding to the third slave node is not the version id with the latest update time.
In some embodiments, the master node determines a third slave node according to the version identification of the data block corresponding to each slave node, and the version identification of the data block corresponding to the third slave node is not the version identification with the latest update time.
Step S807, sending the data corresponding to the version identifier with the latest update time to the third slave node.
In some embodiments, the master node may further determine a third slave node that does not include the version identification corresponding to the most recent update time; and sending the data corresponding to the version identification with the latest updating time to the third slave node.
In some embodiments, after the slave nodes write data successfully, the slave nodes may further send confirmation information to the master node, and the master node marks, according to the confirmation information, the validity of the data block corresponding to the version identifier whose update time is the latest in each slave node.
Therefore, by the data processing method provided by the embodiment of the application, on the premise of ensuring the final consistency, the operation flow in writing data is simplified, when a user writes data, the WAL operation is not required to be executed, the extra performance overhead caused by LOG writing is avoided, the data is directly written into a persistent storage medium, and when a system fault occurs, the final consistency of the data can be ensured through a full recovery mode. The model can maximally utilize the performance of a disk in a distributed storage environment, and improve the writing efficiency so as to deal with the scene with very strict writing delay requirements. Moreover, the replication model based on the Log has complex logic, the state converter is difficult to understand, and the requirement on a code writer is higher. The data processing method provided by the application embodiment has simple logic and clear arrangement, and is easy to convert the model into practical application.
Fig. 8 shows an alternative structural diagram of a master node provided in an embodiment of the present application, which will be described according to various parts.
In some embodiments, master node 900 includes: a receiving unit 901, a transmitting unit 902 and a confirming unit 903. The master node 900 may be a data processing device corresponding to a master node in a cluster, a data processing device corresponding to a cluster, or a data processing device outside a cluster.
The receiving unit 901 is configured to receive and write data to be written sent by a client;
the sending unit 902 is configured to send the data to be written to at least one slave node in the cluster;
the confirming unit 903 is configured to confirm that the data to be written is successfully written into the cluster if first confirmation information sent by a first slave node in all the slave nodes is received; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written.
In some embodiments, the master node 900 may also include a write unit 904.
The writing unit 904 is configured to write the version identifier corresponding to the data to be written.
The confirming unit 903 is specifically configured to confirm that the data block corresponding to the data to be written is valid at the first slave node; and/or confirming that a data block corresponding to the data to be written fails at a second slave node, wherein the second slave node is a slave node except the first slave node in all the slave nodes.
The confirming unit 903 is further configured to obtain a data block corresponding to each slave node; and the master node determines the validity of each data block according to the attribute information of the data block corresponding to each slave node to obtain a confirmation result.
In some embodiments, if the confirmation result indicates that at least one of the data blocks is invalid, the confirming unit 903 is further configured to confirm version identifiers respectively corresponding to the data blocks in all nodes included in the cluster; determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time; if the first node is the master node, determining a third slave node, wherein the version identifier of the data block corresponding to the third slave node is not the version identifier with the latest update time; and sending the data corresponding to the version identification with the latest updating time to the third slave node.
In some embodiments, if the confirmation result indicates that at least one of the data blocks is invalid, the confirming unit 903 is further configured to confirm version identifiers respectively corresponding to the data blocks in all nodes included in the cluster; determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time; if the first node is not the master node, the writing unit 904 is further configured to write a data block corresponding to the version identifier with the latest update time; the master node determines a third slave node, wherein the version identification of the data block corresponding to the third slave node is not the version identification with the latest updating time; and sending the data corresponding to the version identification with the latest updating time to the third slave node.
The sending unit 902 is further configured to send second confirmation information to the client after confirming that the data to be written is successfully written into the cluster; the second acknowledgement information is used to indicate that the cluster successfully writes the data to be written.
Fig. 9 is a schematic diagram of a hardware component structure of a master node according to an embodiment of the present application, where the master node includes: at least one processor 701, a memory 702, and at least one network unit 704. The various components in the master node are coupled together by a bus system 705. It is understood that the bus system 705 is used to enable communications among the components. The bus system 705 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration the various busses are labeled in figure 9 as the bus system 705.
It will be appreciated that the memory 702 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The non-volatile Memory may be ROM, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic random access Memory (FRAM), Flash Memory (Flash Memory), magnetic surface Memory, optical Disc, or Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memory 702 described in embodiments herein is intended to comprise, without being limited to, these and any other suitable types of memory.
The memory 702 in the embodiments of the present application is used to store various types of data to support the operation of the master node. Examples of such data include: any computer program for operating on the master node, such as application 722. A program implementing the method of an embodiment of the present application may be included in the application 722.
The method disclosed in the embodiment of the present application may be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 701. The Processor 701 may be a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 701 may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in the memory 702, and the processor 701 may read the information in the memory 702 and perform the steps of the aforementioned methods in conjunction with its hardware.
In an exemplary embodiment, the master node may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, MPUs, or other electronic components for performing the aforementioned methods.
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to the various embodiments of the present application described in the "exemplary methods" section of this specification, above.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a method according to various embodiments of the present application described in the "exemplary methods" section above of this specification.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (10)
1. A method of data processing, the method comprising:
a main node in a cluster receives and writes data to be written sent by a client;
the master node sends the data to be written to at least one slave node in the cluster;
if the master node receives first confirmation information sent by a first slave node in all the slave nodes, the master node confirms that the data to be written is successfully written into the cluster; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written.
2. The method of claim 1,
and the master node and/or the first slave node writes the version identification corresponding to the data to be written.
3. The method of claim 1, wherein after the master node receives the first acknowledgement information sent by the first slave node of all the slave nodes, the method further comprises:
the master node confirms that a data block corresponding to the data to be written is valid in the first slave node;
and/or the master node confirms that a data block corresponding to the data to be written fails at a second slave node, wherein the second slave node is a slave node except the first slave node in all the slave nodes.
4. The method of claim 3, further comprising:
the master node acquires a data block corresponding to each slave node;
and the master node determines the validity of each data block according to the attribute information of the data block corresponding to each slave node to obtain a confirmation result.
5. The method of claim 4, wherein if the validation result indicates that at least one of the data blocks is invalid, the method further comprises:
the master node confirms the version identifications corresponding to the data blocks in all the nodes included in the cluster respectively;
determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time;
if the first node is the master node, the master node determines a third slave node, and the version identifier of the data block corresponding to the third slave node is not the version identifier with the latest update time;
and sending the data corresponding to the version identification with the latest updating time to the third slave node.
6. The method of claim 4, wherein if the validation result indicates that at least one of the data blocks is invalid, the method further comprises:
the master node confirms the version identifications corresponding to the data blocks in all the nodes included in the cluster respectively;
determining a first node corresponding to a data block corresponding to the version identifier with the latest updating time;
if the first node is not the master node, the master node writes data corresponding to the version identifier with the latest updating time;
the master node determines a third slave node which does not comprise the data block corresponding to the version identifier with the latest updating time, wherein the version identifier of the data block corresponding to the third slave node is not the version identifier with the latest updating time;
and sending the data corresponding to the version identification with the latest updating time to the third slave node.
7. The method of claim 1, wherein after confirming that the data to be written is successfully written into the cluster, the method further comprises:
the main node sends second confirmation information to the client; the second acknowledgement information is used to indicate that the cluster successfully writes the data to be written.
8. A host node, comprising:
the receiving unit is used for receiving and writing data to be written sent by the client;
the sending unit is used for sending the data to be written to at least one slave node in the cluster;
the confirming unit is used for confirming that the data to be written is successfully written into the cluster if first confirming information sent by a first slave node in all the slave nodes is received; the first confirmation information is used for indicating that the first slave node successfully writes the data to be written.
9. An electronic device, comprising:
a memory for storing executable instructions;
a processor for implementing the method steps of any one of claims 1 to 7 when executing executable instructions stored in the memory.
10. A computer-readable write medium, characterized in that a computer program is written in the computer-readable write medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110925774.5A CN113778331B (en) | 2021-08-12 | 2021-08-12 | Data processing method, master node and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110925774.5A CN113778331B (en) | 2021-08-12 | 2021-08-12 | Data processing method, master node and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113778331A true CN113778331A (en) | 2021-12-10 |
| CN113778331B CN113778331B (en) | 2024-06-07 |
Family
ID=78837815
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110925774.5A Active CN113778331B (en) | 2021-08-12 | 2021-08-12 | Data processing method, master node and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113778331B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116910158A (en) * | 2023-08-17 | 2023-10-20 | 深圳计算科学研究院 | Data processing and inquiring method, device, equipment and medium based on copy group |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050237926A1 (en) * | 2004-04-22 | 2005-10-27 | Fan-Tieng Cheng | Method for providing fault-tolerant application cluster service |
| US20120011098A1 (en) * | 2009-03-19 | 2012-01-12 | Murakumo Corporation | Method and system for managing replication of data |
| CN102937964A (en) * | 2012-09-28 | 2013-02-20 | 无锡江南计算技术研究所 | Intelligent data service method based on distributed system |
| US20130198309A1 (en) * | 2011-08-08 | 2013-08-01 | Adobe Systems Incorporated | Clustering Without Shared Storage |
| CN105426439A (en) * | 2015-11-05 | 2016-03-23 | 腾讯科技（深圳）有限公司 | Metadata processing method and device |
| CN106569729A (en) * | 2015-10-09 | 2017-04-19 | 阿里巴巴集团控股有限公司 | Method and device for writing in data in distributed system |
| CN107295080A (en) * | 2017-06-19 | 2017-10-24 | 北京百度网讯科技有限公司 | Date storage method and server applied to distributed server cluster |
| US20180232412A1 (en) * | 2017-02-10 | 2018-08-16 | Sap Se | Transaction commit protocol with recoverable commit identifier |
| CN110231915A (en) * | 2019-05-29 | 2019-09-13 | 南昌大学 | Data managing method, system, device, computer equipment and storage medium |
| WO2019184164A1 (en) * | 2018-03-30 | 2019-10-03 | 平安科技（深圳）有限公司 | Method for automatically deploying kubernetes worker node, device, terminal apparatus, and readable storage medium |
| CN110825309A (en) * | 2018-08-08 | 2020-02-21 | 华为技术有限公司 | Data reading method, device and system and distributed system |
| CN110933137A (en) * | 2019-10-31 | 2020-03-27 | 北京浪潮数据技术有限公司 | Data synchronization method, system, equipment and readable storage medium |
| CN111368002A (en) * | 2020-03-05 | 2020-07-03 | 广东小天才科技有限公司 | Data processing method, system, computer equipment and storage medium |
| CN112134887A (en) * | 2020-09-23 | 2020-12-25 | 哈尔滨海能达科技有限公司 | Data synchronization method and device for nodes in distributed cluster |
| CN112148798A (en) * | 2020-10-10 | 2020-12-29 | 腾讯科技（深圳）有限公司 | Data processing method and device applied to distributed system |
| CN112486932A (en) * | 2020-12-09 | 2021-03-12 | 北京金山云网络技术有限公司 | Data concurrent writing method and distributed data concurrent writing system |
| CN113220693A (en) * | 2021-06-02 | 2021-08-06 | 北京字节跳动网络技术有限公司 | Computing storage separation system, data access method, medium and electronic device thereof |
| CN113220236A (en) * | 2021-05-17 | 2021-08-06 | 北京青云科技股份有限公司 | Data management method, system and equipment |
-
2021
- 2021-08-12 CN CN202110925774.5A patent/CN113778331B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050237926A1 (en) * | 2004-04-22 | 2005-10-27 | Fan-Tieng Cheng | Method for providing fault-tolerant application cluster service |
| US20120011098A1 (en) * | 2009-03-19 | 2012-01-12 | Murakumo Corporation | Method and system for managing replication of data |
| US20130198309A1 (en) * | 2011-08-08 | 2013-08-01 | Adobe Systems Incorporated | Clustering Without Shared Storage |
| CN102937964A (en) * | 2012-09-28 | 2013-02-20 | 无锡江南计算技术研究所 | Intelligent data service method based on distributed system |
| CN106569729A (en) * | 2015-10-09 | 2017-04-19 | 阿里巴巴集团控股有限公司 | Method and device for writing in data in distributed system |
| CN105426439A (en) * | 2015-11-05 | 2016-03-23 | 腾讯科技（深圳）有限公司 | Metadata processing method and device |
| US20180232412A1 (en) * | 2017-02-10 | 2018-08-16 | Sap Se | Transaction commit protocol with recoverable commit identifier |
| CN107295080A (en) * | 2017-06-19 | 2017-10-24 | 北京百度网讯科技有限公司 | Date storage method and server applied to distributed server cluster |
| WO2019184164A1 (en) * | 2018-03-30 | 2019-10-03 | 平安科技（深圳）有限公司 | Method for automatically deploying kubernetes worker node, device, terminal apparatus, and readable storage medium |
| CN110825309A (en) * | 2018-08-08 | 2020-02-21 | 华为技术有限公司 | Data reading method, device and system and distributed system |
| CN110231915A (en) * | 2019-05-29 | 2019-09-13 | 南昌大学 | Data managing method, system, device, computer equipment and storage medium |
| CN110933137A (en) * | 2019-10-31 | 2020-03-27 | 北京浪潮数据技术有限公司 | Data synchronization method, system, equipment and readable storage medium |
| CN111368002A (en) * | 2020-03-05 | 2020-07-03 | 广东小天才科技有限公司 | Data processing method, system, computer equipment and storage medium |
| CN112134887A (en) * | 2020-09-23 | 2020-12-25 | 哈尔滨海能达科技有限公司 | Data synchronization method and device for nodes in distributed cluster |
| CN112148798A (en) * | 2020-10-10 | 2020-12-29 | 腾讯科技（深圳）有限公司 | Data processing method and device applied to distributed system |
| CN112486932A (en) * | 2020-12-09 | 2021-03-12 | 北京金山云网络技术有限公司 | Data concurrent writing method and distributed data concurrent writing system |
| CN113220236A (en) * | 2021-05-17 | 2021-08-06 | 北京青云科技股份有限公司 | Data management method, system and equipment |
| CN113220693A (en) * | 2021-06-02 | 2021-08-06 | 北京字节跳动网络技术有限公司 | Computing storage separation system, data access method, medium and electronic device thereof |
Non-Patent Citations (1)
| Title |
|---|
| 叶斌等: "基于分布式的远程复制系统研究", 微处理机, no. 04, pages 56 - 60 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116910158A (en) * | 2023-08-17 | 2023-10-20 | 深圳计算科学研究院 | Data processing and inquiring method, device, equipment and medium based on copy group |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113778331B (en) | 2024-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10831741B2 (en) | Log-shipping data replication with early log record fetching | |
| WO2019141186A1 (en) | Data processing method and device | |
| US8868492B2 (en) | Method for maximizing throughput and minimizing transactions response times on the primary system in the presence of a zero data loss standby replica | |
| US8463746B2 (en) | Method and system for replicating data | |
| US11233874B2 (en) | Ordinary write in distributed system maintaining data storage integrity | |
| US8335766B2 (en) | Flash-copying within asynchronous mirroring environment | |
| JP2007272648A (en) | Database system operating method, database system, database device and backup program | |
| US10761748B2 (en) | Future write in distributed system maintaining data storage integrity | |
| WO2021120880A1 (en) | Data replication processing method and apparatus, disaster recovery system, device and storage medium | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| CN114063883B (en) | Data storage method, electronic device and computer program product | |
| US9043283B2 (en) | Opportunistic database duplex operations | |
| WO2019109256A1 (en) | Log management method, server and database system | |
| CN113778331B (en) | Data processing method, master node and storage medium | |
| WO2022134638A1 (en) | Logic clock synchronization method and apparatus, and central time service cluster | |
| US10169441B2 (en) | Synchronous data replication in a content management system | |
| CN109992447B (en) | Data copying method, device and storage medium | |
| US20120191645A1 (en) | Information processing apparatus and database system | |
| CN115698955A (en) | Fault tolerance of transaction images | |
| CN113760862A (en) | Incremental data breakpoint continuous transmission method, device, equipment and storage medium | |
| US20200167264A1 (en) | Multi-level debugger | |
| CN111400404A (en) | Node initialization method, device, equipment and storage medium | |
| US20240248635A1 (en) | Method, electronic device, and computer program product for storing metadata | |
| CN117874129A (en) | Semi-synchronous method for cache data | |
| WO2023030013A1 (en) | Data processing method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |