CN114064349B - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114064349B CN114064349B CN202010757060.3A CN202010757060A CN114064349B CN 114064349 B CN114064349 B CN 114064349B CN 202010757060 A CN202010757060 A CN 202010757060A CN 114064349 B CN114064349 B CN 114064349B
- Authority
- CN
- China
- Prior art keywords
- resource
- cluster
- event
- information
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 230000001360 synchronised effect Effects 0.000 claims description 70
- 238000012544 monitoring process Methods 0.000 claims description 50
- 238000000034 method Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 11
- 238000013507 mapping Methods 0.000 claims 2
- 238000011084 recovery Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000006243 chemical reaction Methods 0.000 description 6
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 3
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 3
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012300 Sequence Analysis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Abstract
The embodiment of the invention provides a data processing method, which is characterized in that resource information on a first cluster is monitored in real time, wherein the first cluster represents a main cluster based on container resources; determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on the container resource; in the embodiment of the invention, the resource information on the first cluster can be monitored in real time, and all resource events occurring on the first cluster can be monitored, so that the resource information loss caused by the backup time interval is avoided. The embodiment of the invention also provides a data processing device, electronic equipment and a computer storage medium.
Description
Technical Field
The present invention relates to data processing technology, and in particular, to a data processing method, apparatus, electronic device, and computer storage medium.
Background
In the related art, the disaster recovery backup method across Kubernetes clusters mainly realizes disaster recovery backup between Kubernetes clusters by backing up data of Etcd components in the Kubernetes clusters, so that snapshot backup is required to be performed on Etcd components regularly, and whether the Etcd components of the Kubernetes clusters are manually backed up to recover the failed Kubernetes clusters or the failed Kubernetes clusters are recovered by a synchronization tool, the data change information generated by data operation in a time interval of two backups may not be backed up, so that the data change information is lost.
Disclosure of Invention
The embodiment of the invention expects to provide a technical scheme for data processing, so as to solve the problem that the data change information is lost because the data change information generated by data operation in the time interval of two backups cannot be backed up in the related technology.
The embodiment of the invention provides a data processing method, which comprises the following steps:
Monitoring resource information on a first cluster in real time, wherein the first cluster represents a main cluster based on container resources;
Determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on the container resource.
Optionally, the synchronizing the at least one resource event to the second cluster includes:
and under the condition that the number of the at least one resource event is larger than 1, synchronizing the at least one resource event to a second cluster according to the occurrence sequence of the at least one resource event.
Optionally, the synchronizing the at least one resource event to the second cluster according to the sequence of occurrence of the at least one resource event includes:
storing the at least one resource event in a message queue according to the sequence of occurrence of the at least one resource event;
And acquiring the at least one resource event from the message queue according to the sequence, and synchronizing the acquired at least one resource event to the second cluster.
Optionally, the monitoring, in real time, the resource information on the first cluster includes:
determining a synchronous resource type, wherein the synchronous resource type represents the type of the resource needing to be monitored on the first cluster;
And monitoring the resource information corresponding to the synchronous resource type in real time.
Optionally, the method further comprises:
and under the condition that the at least one resource event is successfully synchronized to the second cluster, updating the synchronization progress information, wherein the synchronization progress information represents the synchronization state information corresponding to the at least one resource event.
Optionally, the monitoring the resource information on the first cluster in real time includes: the resource information on the first cluster is monitored using the synchronization component,
The method further comprises the steps of:
And under the condition that the synchronous component is restarted, acquiring the synchronous progress information, and determining a resource event needing to be synchronized according to the synchronous progress information.
Optionally, the updating the synchronization progress information includes: updating the synchronization progress information into a storage database,
The obtaining the synchronization progress information includes:
And acquiring the synchronous progress information from the storage database.
Optionally, the method further comprises:
And controlling the synchronization component to restart in case of failure to synchronize the at least one resource event to the second cluster.
The embodiment of the invention also provides a data processing device, which comprises: a monitoring module and a synchronization module, wherein,
The monitoring module is used for monitoring resource information on a first cluster in real time, wherein the first cluster represents a main cluster based on container resources;
the synchronization module is configured to determine whether to generate at least one resource event according to the resource information, and synchronize the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, where the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the primary cluster based on container resources.
Optionally, the synchronization module is configured to synchronize the at least one resource event to a second cluster, including:
and under the condition that the number of the at least one resource event is larger than 1, synchronizing the at least one resource event to a second cluster according to the occurrence sequence of the at least one resource event.
Optionally, the synchronization module is configured to synchronize the at least one resource event to a second cluster according to the sequence of occurrence of the at least one resource event, including:
storing the at least one resource event in a message queue according to the sequence of occurrence of the at least one resource event;
And acquiring the at least one resource event from the message queue according to the sequence, and synchronizing the acquired at least one resource event to the second cluster.
Optionally, the monitoring module is configured to monitor, in real time, resource information on the first cluster, including:
determining a synchronous resource type, wherein the synchronous resource type represents the type of the resource needing to be monitored on the first cluster;
And monitoring the resource information corresponding to the synchronous resource type in real time.
Optionally, the synchronization module is further configured to update synchronization progress information, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event, when the at least one resource event is determined to be successfully synchronized to the second cluster.
Optionally, the monitoring module is configured to monitor, in real time, resource information on the first cluster, including: and the monitoring module is also used for acquiring the synchronization progress information under the condition that the synchronization component is restarted, and determining resource events needing to be synchronized according to the synchronization progress information.
Optionally, the synchronization module is configured to update synchronization progress information, including: updating the synchronization progress information into a storage database,
The monitoring module is used for acquiring the synchronization progress information, and comprises:
And acquiring the synchronous progress information from the storage database.
Optionally, the monitoring module is further configured to control the synchronization component to restart in case of failure to synchronize the at least one resource event to the second cluster.
The embodiment of the invention also provides electronic equipment, which comprises a processor and a memory for storing a computer program capable of running on the processor; wherein,
The processor is configured to execute any one of the data processing methods described above when running the computer program.
The embodiment of the invention also provides a computer storage medium, on which a computer program is stored, which when executed by a processor implements any of the above described data processing methods.
It can be seen that, in the data processing method in the embodiment of the present invention, by monitoring resource information on a first cluster in real time, the first cluster represents a main cluster based on container resources; determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources; in the embodiment of the invention, the resource information on the first cluster can be monitored in real time, and all resource events occurring on the first cluster can be monitored, so that the resource information loss caused by the backup time interval is avoided.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a Kubernetes resource backup architecture according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the internal architecture of a synchronization assembly according to an embodiment of the present invention;
FIG. 4 is a schematic workflow diagram of a synchronization component according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples. It is to be understood that the examples provided herein are for the purpose of illustration only and are not intended to limit the invention. In addition, the embodiments provided below are some of the embodiments for carrying out the present invention, but not all of the embodiments for carrying out the present invention, and the technical solutions described in the embodiments of the present invention may be implemented in any combination without conflict.
It should be noted that, in the embodiments of the present invention, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a method or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such method or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other related elements in a method or apparatus comprising the element (e.g., a step in a method or an element in an apparatus, e.g., an element may be part of a circuit, part of a processor, part of a program or software, etc.).
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, may mean including any one or more elements selected from the group consisting of A, B and C.
For example, the data processing method provided in the embodiment of the present invention includes a series of steps, but the data processing method provided in the embodiment of the present invention is not limited to the described steps, and similarly, the data processing apparatus provided in the embodiment of the present invention includes a series of modules, but the apparatus provided in the embodiment of the present invention is not limited to the explicitly described modules, and may also include modules that are required to be set for acquiring relevant information or performing processing based on the information.
Embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations, and with computer executable code executed by a processor. Here, the server may be a server computer system small computer system, a large computer system, a distributed cloud computing technology environment including any of the above, and the like.
A server may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The servers may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.
As the demand for containerization increases, the wider the use of Kubernetes as a container orchestration engine, and the high availability and disaster recovery in a production environment becomes a functional point that has to be considered. The disaster recovery mode for containers in Kubernetes is mainly realized by restarting Pod through a multi-copy mode or probe detection inside a Kubernetes cluster. For disaster recovery modes among different Kubernetes clusters, in the related art, a common disaster recovery mode is to snapshot Etcd components in the Kubernetes, and when the Kubernetes main cluster cannot work normally, the disaster recovery mode can be manually restored to the standby Kubernetes cluster by snapshot Etcd components or by a synchronization tool of Etcd components, for example, ETCDCTL MAKE-minor, so as to realize synchronization of data in the Etcd components.
For the above-mentioned manner of disaster recovery in a Kubernetes cluster, only the disaster recovery in the case that the main node of the Kubernetes cluster can work normally, but the service container cannot work normally is supported, and when the main node of the Kubernetes cluster cannot work, the management component on the main node cannot work, and thus the container on the service node cannot be managed, so that when the service container has a problem, the problem of the service cluster cannot be recovered.
For the above disaster recovery backup manner of backing up the data of Etcd components in Kubernetes clusters, since the manner needs to periodically perform snapshot backup on Etcd components in Kubernetes clusters, whether manually backing up Etcd components of Kubernetes clusters to recover failed Kubernetes clusters or recovering failed Kubernetes clusters through a synchronization tool, it may not be possible to backup the data change information generated by the data operation in the time interval of two backups, which results in loss of the data change information.
In order to solve the above technical problems, an embodiment of the present invention provides a data processing method, and fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the flowchart may include:
step 101: resource information on a first cluster representing a primary cluster based on container resources is monitored in real-time.
In one example, a container resource may be represented as a resource unit in units of resource containers, where the computing resources of the system may be uniformly divided and packaged into a number of resource units.
In one embodiment, the first cluster may be a Kubernetes master cluster, or may be another type of master cluster based on container resources, and the resource information may represent change information of resources in the Kubernetes cluster. The implementation manner of monitoring the resource information on the first cluster in real time may be to determine a synchronous resource type, where the synchronous resource type represents a type of a resource to be monitored on the first cluster, and further, the resource information corresponding to the synchronous resource type is monitored in real time.
In one example, the synchronization resource type may include one resource type or multiple resource types, for example, the synchronization resource type may include ConfigMap one resource type or three resource types, configMap, depleyment and Namespace. The implementation manner of determining the type of the synchronization resource may be determining the resource to be synchronized on the first cluster according to the user requirement, determining the corresponding type of the resource according to the determined resource to be synchronized, for example, in the case that the user requirement is to synchronize ConfigMap and DEPLEYMENT resources, the determined corresponding type of the resource is two types of ConfigMap and DEPLEYMENT, meanwhile, when determining the resource to be synchronized on the first cluster, a synchronization component may be created, and the synchronization component may be a SyncComponent component, further, the synchronization component may be considered as a custom resource (Custom Resource Definition, CRD) of Kubernetes, where the CRD of the synchronization component may be defined as:
After the CRD of the synchronization component is created, a control component (Controller) corresponding to the CRD monitors, and creates a service instance corresponding to the synchronization component, where the control component (Controller) is located in the Kubernetes cluster.
When the second cluster represents the backup cluster corresponding to the first cluster and the second cluster need to be synchronized, the following related settings need to be performed in the synchronization component:
The name of the synchronization component SyncComponent may be set as a unique identifier of the second cluster (backup cluster), and setting information corresponding to the spec includes setting information such as unique identifiers of the first cluster (primary cluster) and the second cluster (backup cluster) and related certificates of the second cluster (backup cluster), and through the setting of these information, the synchronization component may operate the first cluster and the second cluster simultaneously.
For the implementation manner of monitoring the resource information corresponding to the synchronous resource type in real time, the monitoring module (Watcher) in the synchronous component can be used for monitoring the resource information corresponding to the synchronous resource type in real time, specifically, the monitoring module (Watcher) can monitor the change condition of various resource information on the first cluster, convert the data corresponding to the resource change condition into data in a yaml format, and store the converted data in a message queue in the synchronous component SyncComponent; specifically, the real-time monitoring of the resource information can be achieved through two different monitoring modules (Watcher) corresponding to two resource types ConfigMap and DEPLEYMENT, for example, for the ConfigMap resource corresponding monitoring module Watcher1 and for the DEPLEYMENT resource corresponding monitoring module DEPLEYMENT 2.
It can be seen that the resource type corresponding to the resource requiring the resource synchronization is determined according to the requirement, and further, only the mode of monitoring the resource information corresponding to the determined resource type can be used without monitoring the whole Kubernetes cluster, so that the loss of resource monitoring is reduced.
Step 102: determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on the container resource.
As an embodiment, a resource operation may be an operation of deleting, creating, modifying, etc. a certain resource, e.g., configMap, namespace or DEPLEYMENT resource, and is not specifically limited herein; the resource event may represent resource change information resulting from creating one ConfigMap or DEPLEYMENT resource instance, in particular a certain type of resource may contain a large number of resource instances of that type.
The resource event may be resource change information caused by an operation on one resource instance in the type of resource, and of course, the resource event may also be resource change information caused by an operation on 1 resource instance in the multiple types of resource; specifically, for ConfigMap resources, the resources may include a plurality of ConfigMap resource instances, for example, may include a resource instance a, a resource instance B, a resource instance C, and a resource instance D, where a resource event may be information about a change in a resource caused by deleting the resource instance a, or may be information about a change in a resource caused by creating the resource instance B; for ConfigMap resources and DEPLEYMENT resources, and DEPLEYMENT resources include resource instance E, resource instance F, resource instance G, and resource instance H, the resource event may be any of deleting resource instance a, modifying resource instance B, creating resource instance G, or deleting resource F.
In one example, the determining whether to generate at least one resource event according to the resource information may be determining whether to generate the resource event 1 and the resource event 2 according to a detected change of the resource information caused by deleting the resource instance a and modifying the resource instance B. In case it is determined to generate at least one resource event, the implementation of synchronizing the at least one resource event to the second cluster may be that in case it is determined to generate at least one resource event, the generated resource event 1 and the generated resource event 2 are synchronized to the second cluster.
In practical applications, steps 101 to 102 may be implemented by a Processor in an electronic device, where the Processor may be at least one of an Application SPECIFIC INTEGRATED Circuit (ASIC), a digital signal Processor (DIGITAL SIGNAL Processor, DSP), a digital signal processing device (DIGITAL SIGNAL Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), an FPGA, a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor.
It can be seen that the embodiment of the invention provides a data processing method, by monitoring resource information on a first cluster in real time, the first cluster represents a main cluster based on container resources; determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on the container resource; in the embodiment of the invention, the resource information on the first cluster can be monitored in real time, and all resource events occurring on the first cluster can be monitored, so that the resource information loss caused by the backup time interval is avoided.
In one embodiment, the synchronizing the at least one resource event to the second cluster includes: and under the condition that the number of the at least one resource event is larger than 1, synchronizing the at least one resource event to a second cluster according to the occurrence sequence of the at least one resource event.
In one example, the at least one resource event is synchronized to the second cluster according to the sequence of occurrence of the at least one resource event, which may be that the at least one resource event is stored in a message queue according to the sequence of occurrence of the at least one resource event, the at least one resource event is obtained from the message queue according to the sequence of occurrence of the at least one resource event, and the obtained at least one resource event is synchronized to the second cluster.
Specifically, for the case that the at least one resource event is a resource event 1 and a resource event 2, and the resource event 1 occurs before the resource event 2, the implementation manner of storing the at least one resource event in the message queue according to the sequence of the at least one resource event may be that the resource event 1 is stored in the message queue and then the resource event 2 is stored in the message queue, the at least one resource event is acquired from the message queue according to the sequence, and the acquired at least one resource event is synchronized to the second cluster, or that the resource event 1 is acquired from the message queue and the resource event 1 is synchronized to the second cluster, and then the resource event 2 is acquired from the message queue and the resource event 2 is synchronized to the second cluster. Here, the message queue may be located in the synchronization component and used to store various resource events to ensure that the events can be executed in sequence. The synchronous component also comprises
In an example, the at least one resource event may be synchronized to the second cluster by the processing module Handler in the synchronization component obtaining at least one resource event in the message queue and synchronizing the obtained at least one resource event to the second cluster, and specifically, the processing module Handler may obtain data in yaml format in the message queue and deserialize the obtained data in yaml format, parse out the corresponding resource event, and synchronize the second cluster according to the parsed resource event.
The implementation manner of synchronizing the at least one resource event to the second cluster may, for example, determine a specific resource instance corresponding to the resource event for the resource event, and identify the determined resource instance, where the processing module Handler performs synchronization operation only on the identified resource instance and not on the unidentified resource instance, for example, for the case where the at least one resource event is resource event 1 and resource event 2, respectively, where resource event 1 represents resource change information caused by deleting ConfigMap resource instance a, resource event 2 represents resource change information caused by creating DEPLEYMENT resource instance B, and resource event 1 occurs before resource event 2, and identifies ConfigMap resource instance a and DEPLEYMENT resource instance B, respectively, but does not identify other ConfigMap resource instances and DEPLEYMENT resource instances other than ConfigMap resource instance a and DEPLEYMENT resource instance B, and the processing module Handler performs synchronization operation only on identified ConfigMap resource instance a and DEPLEYMENT resource instance B, and not on other DEPLEYMENT resource instances other than unidentified ConfigMap resource instance a and DEPLEYMENT resource instance ConfigMap.
It can be seen that, by the above synchronization method, granularity of backup can be reduced to a resource instance level, all resources in the whole first cluster (Kubernetes main cluster) do not need to be backed up, specific synchronization resources can be determined according to user requirements to synchronize, synchronization logic of other resources cannot be affected, synchronization is enabled to be more flexible, and customizable backup can be performed for different types of resources through specific implementation of the processing module Handle, so that resource loss in the backup process is reduced.
In one embodiment, in a case that the at least one resource event is successfully synchronized to the second cluster, synchronization progress information may be updated, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event. The updating of the synchronization progress information includes: and updating the synchronous progress information into a storage database.
In one example, the synchronization progress information may be relationship information used to indicate before and after the occurrence of the resource event, specifically, the synchronization progress information may be a version number of the resource event, for example, the first synchronization information may be a version number 1 and a version number 2, where the resource event corresponding to the version number 1 occurs before the resource event corresponding to the version number 2, and the storage database may be a Etcd component, where the Etcd component is a service discovery storage database with high availability and strong consistency, and may be responsible for saving the current synchronization states of various resources.
In one example, in the case that the at least one resource event is determined to be successfully synchronized to the second cluster, the synchronization progress information may be updated for the first time after determining that the first resource event is successfully synchronized to the second cluster, and after determining that the second resource event is successfully synchronized to the second cluster, the synchronization progress information is updated for the second time, for example, for the resource event a and the resource event B, where the resource event a occurs before the resource event B, the version number 1 may be updated to the version number 2 after the resource event a is successfully updated to the second cluster, and the version number 2 may be updated to the version number 3 after the resource event B is successfully updated to the second cluster.
The implementation of updating the synchronization progress information into the storage database may be to update the version number of the resource event into the storage database, for example, update the version number 1 into the data repository.
In one embodiment, under the condition that the synchronization component is restarted, the synchronization progress information is obtained, and a resource event needing to be synchronized is determined according to the synchronization progress information. The step of obtaining the synchronization progress information may be obtaining the synchronization progress information from the storage database.
In an example, under the condition that the synchronization component is restarted, the synchronization progress information is obtained, a resource event needing to be synchronized is determined according to the synchronization progress information, that is, after the synchronization component is restarted due to failure, a latest version number of the resource event is obtained from a storage database, a resource synchronization state is determined according to the obtained latest version number, and then the resource event needing to be synchronized is determined. Specifically, the first synchronization information corresponding to the resource event a is version number 5, and the resource event a fails to synchronize the resource event a to the second cluster due to the failure of the synchronization component, at this time, the version number in the storage database should be version number 4 corresponding to the last resource event corresponding to the resource event a of the second cluster, and the message queue in the corresponding synchronization component does not delete the data corresponding to the resource event a which is not successfully synchronized to the second cluster, after the synchronization component is restarted, the latest version number 4 of the resource event is acquired from the Etcd component, and the resource event a is acquired from the message queue again through the processing module Handler, and is synchronized to the second cluster, and meanwhile, the monitored resource event after the synchronization component is restarted is sequentially synchronized, so that the synchronization event can be continuously synchronized according to the synchronization progress after the restart, and repeated synchronization of the same resource event cannot occur.
In one embodiment, the synchronization component may be controlled to restart in the event that synchronization of the at least one resource event to the second cluster fails.
In one embodiment, when the synchronization of the resource event to the second cluster fails, the synchronization component may be controlled to restart according to the reason of the synchronization failure, where the solution corresponding to the reason of the failure is to restart the synchronization component. In particular, an error notification may be issued when the synchronization of the resource event to the second cluster fails, and the specific form of issuing the error notification is not limited herein, and the error notification may be performed by a voice prompt, a dialog box display, or other manners.
Fig. 2 is a schematic diagram of a Kubernetes resource backup architecture according to an embodiment of the present invention, as shown in fig. 2, service clusters in a primary data center are all primary clusters, service clusters in a backup data center are all backup clusters, and the primary clusters in the primary data center respectively correspond to the backup clusters in the backup data center, and a synchronization component may monitor resource information on the primary service clusters and implement a resource synchronization operation on the backup clusters.
Fig. 3 is a schematic diagram of an internal architecture of a synchronization component according to an embodiment of the present invention, where the synchronization component, as shown in fig. 3, is composed of a monitoring module, a message queue and a processing module, where the monitoring module may read Etcd synchronization progress information in the component, monitor resource information in a primary cluster, and when determining to generate a resource event according to the monitored resource information, perform format conversion on the determined resource event, and store the resource event after format conversion in the internal message queue, the processing module may obtain the resource event after format conversion in the message queue, perform inverse sequence analysis on the resource event after format conversion, synchronize the corresponding resource event to a backup cluster according to an analysis result, and update Etcd synchronization progress information in the component under the condition that the resource event is successfully synchronized to the backup cluster.
Fig. 4 is a schematic workflow diagram of a synchronization component according to an embodiment of the present invention, as shown in fig. 4, when the synchronization component starts, a corresponding resource type is determined according to a resource to be synchronized, a monitoring module and a processing module corresponding to each resource type are initialized respectively according to a type of the resource type, after the monitoring module starts, synchronization progress information can be obtained from the Etcd component and resource information corresponding to resources to be synchronized in a main cluster is monitored, when a resource event is determined to be generated according to the monitored resource information, format conversion is performed on the generated resource event, the resource event after format conversion is placed in a message queue, then the monitoring module waits for the occurrence of a next resource event, when a monitoring error occurs in the monitoring module, the synchronization progress is re-obtained from the Etcd component and starts to monitor for a new time, the processing module starts to wait for the resource event in the message queue after the monitoring module starts, after the resource event is obtained, reverse-sequence analysis is performed on the resource event, and the backed up resource event is synchronized to the cluster, when the resource event is successfully synchronized to the backup cluster, the synchronization progress information is updated, and the synchronization progress information is not updated to the synchronization information is updated to the Etcd, and the synchronization progress of the synchronization component can be continued.
On the basis of the data processing method provided by the foregoing embodiment, an embodiment of the present invention provides a data processing apparatus.
Fig. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 5, the apparatus may include: a monitor module 501 and a synchronization module 502, wherein,
The monitoring module 501 is configured to monitor resource information on a first cluster in real time, where the first cluster represents a primary cluster based on container resources;
The synchronization module 502 is configured to determine whether to generate at least one resource event according to the resource information, and synchronize the at least one resource event to a second cluster in the case that it is determined to generate the at least one resource event, where the resource event represents resource change information caused by a resource operation, and the second cluster represents a backup cluster corresponding to the primary cluster based on the container resource.
Optionally, the synchronizing module 502 is configured to synchronize the at least one resource event to a second cluster, including:
and under the condition that the number of the at least one resource event is larger than 1, synchronizing the at least one resource event to a second cluster according to the occurrence sequence of the at least one resource event.
Optionally, the synchronization module 502 is configured to synchronize the at least one resource event to the second cluster according to the sequence of occurrence of the at least one resource event, including:
storing the at least one resource event in a message queue according to the sequence of occurrence of the at least one resource event;
And acquiring the at least one resource event from the message queue according to the sequence, and synchronizing the acquired at least one resource event to the second cluster.
Optionally, the monitoring module 501 is configured to monitor, in real time, resource information on the first cluster, including:
determining a synchronous resource type, wherein the synchronous resource type represents the type of the resource needing to be monitored on the first cluster;
And monitoring the resource information corresponding to the synchronous resource type in real time.
Optionally, the synchronization module 502 is further configured to update synchronization progress information, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event, in a case where the at least one resource event is successfully synchronized to the second cluster.
Optionally, the monitoring module 501 is configured to monitor, in real time, resource information on the first cluster, including: and the monitoring module is also used for acquiring the synchronization progress information under the condition that the synchronization component is restarted, and determining resource events needing to be synchronized according to the synchronization progress information.
Optionally, the synchronization module 502 is configured to update synchronization progress information, including: updating the synchronization progress information into a storage database,
The monitoring module is used for acquiring the synchronization progress information, and comprises:
And acquiring the synchronous progress information from the storage database.
Optionally, the monitoring module 501 is further configured to control the synchronization component to restart in case of failure to synchronize the at least one resource event to the second cluster.
In practical applications, the monitoring module 501 and the synchronization module 502 may be implemented by a processor in an electronic device, where the processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor.
In addition, each functional module in the present embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional modules.
The integrated units, if implemented in the form of software functional modules, may be stored in a computer-readable storage medium, where the technical solution of the present embodiment is essentially or partly contributing to the related art or all or part of the technical solution may be embodied in the form of a software product, which is stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform all or part of the steps of the method described in the present embodiment. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Specifically, the computer program instructions corresponding to one data processing method in the present embodiment may be stored on a storage medium such as an optical disc, a hard disk, or a usb disk, and when the computer program instructions corresponding to one video processing method in the storage medium are read or executed by an electronic device, any one of the data processing methods in the foregoing embodiments is implemented.
Based on the same technical concept as the foregoing embodiments, referring to fig. 6, an electronic device provided by an embodiment of the present invention may include: a memory 601 and a processor 602; wherein,
The memory 601 is used for storing computer programs and data;
The processor 602 is configured to execute a computer program stored in the memory to implement any one of the data processing methods of the foregoing embodiments.
In practical applications, the memory 601 may be a volatile memory (RAM); or a non-volatile memory (non-volatile memory), such as ROM, flash memory (flash memory), hard disk (HARD DISK DRIVE, HDD) or Solid state disk (Solid-state-STATE DRIVE, SSD); or a combination of the above types of memory and provides instructions and data to the processor 602.
The processor 602 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor. It will be appreciated that the electronic device for implementing the above-described processor function may be other for different electronic apparatuses, and embodiments of the present invention are not specifically limited.
In some embodiments, the functions or modules included in the apparatus provided by the embodiments of the present invention may be used to perform the methods described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.
The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.
The methods disclosed in the method embodiments provided by the application can be arbitrarily combined under the condition of no conflict to obtain a new method embodiment.
The features disclosed in the embodiments of the products provided by the application can be combined arbitrarily under the condition of no conflict to obtain new embodiments of the products.
The features disclosed in the embodiments of the method or the device provided by the application can be arbitrarily combined under the condition of no conflict to obtain a new embodiment of the method or the device.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the related art in the form of a software product stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk), including several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (11)
1. A method of data processing, the method comprising:
Monitoring resource information on a first cluster in real time, wherein the first cluster represents a main cluster based on container resources; the resource information is information corresponding to a synchronous resource type, wherein the synchronous resource type comprises configuration mapping ConfigMap, deployment DEPLEYMENT and a Namespace;
Determining whether to generate at least one resource event according to the resource information, and synchronizing the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, wherein the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the main cluster based on the container resource.
2. The method of claim 1, wherein synchronizing the at least one resource event to a second cluster comprises:
and under the condition that the number of the at least one resource event is larger than 1, synchronizing the at least one resource event to a second cluster according to the occurrence sequence of the at least one resource event.
3. The method of claim 2, wherein synchronizing the at least one resource event to the second cluster in the order in which the at least one resource event occurred comprises:
storing the at least one resource event in a message queue according to the sequence of occurrence of the at least one resource event;
And acquiring the at least one resource event from the message queue according to the sequence, and synchronizing the acquired at least one resource event to the second cluster.
4. The method of claim 1, wherein the monitoring the resource information on the first cluster in real time comprises:
determining a synchronous resource type, wherein the synchronous resource type represents the type of the resource needing to be monitored on the first cluster;
And monitoring the resource information corresponding to the synchronous resource type in real time.
5. The method according to claim 1, wherein the method further comprises:
and under the condition that the at least one resource event is successfully synchronized to the second cluster, updating the synchronization progress information, wherein the synchronization progress information represents the synchronization state information corresponding to the at least one resource event.
6. The method of claim 5, wherein monitoring the resource information on the first cluster in real time comprises: the resource information on the first cluster is monitored using the synchronization component,
The method further comprises the steps of:
And under the condition that the synchronous component is restarted, acquiring the synchronous progress information, and determining a resource event needing to be synchronized according to the synchronous progress information.
7. The method of claim 6, wherein the updating of the synchronization progress information comprises: updating the synchronization progress information to a storage database;
the obtaining the synchronization progress information includes:
And acquiring the synchronous progress information from the storage database.
8. The method of claim 6, wherein the method further comprises:
And controlling the synchronization component to restart in case of failure to synchronize the at least one resource event to the second cluster.
9. A data processing apparatus, the apparatus comprising: a monitoring module and a synchronization module, wherein,
The monitoring module is used for monitoring resource information on a first cluster in real time, wherein the first cluster represents a main cluster based on container resources; the resource information is information corresponding to a synchronous resource type, wherein the synchronous resource type comprises configuration mapping ConfigMap, deployment DEPLEYMENT and a Namespace;
the synchronization module is configured to determine whether to generate at least one resource event according to the resource information, and synchronize the at least one resource event to a second cluster under the condition that the at least one resource event is determined to be generated, where the resource event represents resource change information caused by resource operation, and the second cluster represents a backup cluster corresponding to the primary cluster based on container resources.
10. An electronic device comprising a processor and a memory for storing a computer program capable of running on the processor; wherein,
The processor being arranged to execute the data processing method of any of claims 1 to 8 when the computer program is run.
11. A computer storage medium on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the data processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010757060.3A CN114064349B (en) | 2020-07-31 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010757060.3A CN114064349B (en) | 2020-07-31 | Data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114064349A CN114064349A (en) | 2022-02-18 |
| CN114064349B true CN114064349B (en) | 2024-11-15 |
Family
ID=
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110572464A (en) * | 2019-09-12 | 2019-12-13 | 北京浪潮数据技术有限公司 | Bottom layer resource state synchronization method and device, cloud platform system and storage medium |
| CN110633322A (en) * | 2019-09-12 | 2019-12-31 | 北京浪潮数据技术有限公司 | Resource information synchronization method and device, electronic equipment and storage medium |
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110572464A (en) * | 2019-09-12 | 2019-12-13 | 北京浪潮数据技术有限公司 | Bottom layer resource state synchronization method and device, cloud platform system and storage medium |
| CN110633322A (en) * | 2019-09-12 | 2019-12-31 | 北京浪潮数据技术有限公司 | Resource information synchronization method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020072338A1 (en) | Data backup and disaster recovery between environments | |
| US9880827B2 (en) | Managing software version upgrades in a multiple computer system environment | |
| RU2463646C2 (en) | Dynamic command line interface (cli) mapping for clustered software entities | |
| CN113569987A (en) | Model training method and device | |
| CN109871384B (en) | Method, system, equipment and storage medium for container migration based on PaaS platform | |
| CN106657167B (en) | Management server, server cluster, and management method | |
| CN109558260B (en) | Kubernetes fault elimination system, method, equipment and medium | |
| US7698391B2 (en) | Performing a provisioning operation associated with a software application on a subset of the nodes on which the software application is to operate | |
| JP4141875B2 (en) | Recovery processing method, its execution system, and its processing program | |
| CN105589756B (en) | Batch processing group system and method | |
| WO2017028375A1 (en) | Version upgrading method and system | |
| CN113722155A (en) | Data backup and repair method in distributed file system and related components | |
| US20120239975A1 (en) | Reducing Application Downtime During Failover | |
| CN114064349B (en) | Data processing method, device, equipment and storage medium | |
| CN111342986A (en) | Distributed node management method and device, distributed system and storage medium | |
| CN114064438A (en) | Database fault processing method and device | |
| CN114237510A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN118018463A (en) | Fault processing method, device, equipment and readable storage medium | |
| WO2024041363A1 (en) | Serverless-architecture-based distributed fault-tolerant system, method and apparatus, and device and medium | |
| US20240211157A1 (en) | Automatic data mover selection in information processing system environment | |
| CN117573306A (en) | Batch task scheduling system, method, device, computer equipment and storage medium | |
| CN114064349A (en) | Data processing method, device, equipment and storage medium | |
| CN106209450B (en) | Server fault switching method and application automation deployment system | |
| CN116319758A (en) | Data migration method, device, electronic equipment and readable storage medium | |
| CN102281159A (en) | Recovery method for cluster system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |