Nothing Special   »   [go: up one dir, main page]

WO2024021470A1 - Cross-region data scheduling method and apparatus, device, and storage medium - Google Patents

Cross-region data scheduling method and apparatus, device, and storage medium Download PDF

Info

Publication number
WO2024021470A1
WO2024021470A1 PCT/CN2022/141408 CN2022141408W WO2024021470A1 WO 2024021470 A1 WO2024021470 A1 WO 2024021470A1 CN 2022141408 W CN2022141408 W CN 2022141408W WO 2024021470 A1 WO2024021470 A1 WO 2024021470A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
information
data center
target
metadata
Prior art date
Application number
PCT/CN2022/141408
Other languages
French (fr)
Chinese (zh)
Inventor
颜嘉伟
黄润怀
林洁琬
肖露
吴文峰
陈杰
张维杰
Original Assignee
天翼云科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 天翼云科技有限公司 filed Critical 天翼云科技有限公司
Publication of WO2024021470A1 publication Critical patent/WO2024021470A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present application relates to the field of cloud computing technology, and in particular to a cross-regional data scheduling method, device, equipment and storage medium.
  • SAN Storage Area Network
  • servers hosts
  • storage devices logical disk units (Logical Disk Units, LUNs) or disk arrays
  • bridges and multiplexers are a network-centric storage structure that usually includes servers (hosts), storage devices (logical disk units (Logical Disk Units, LUNs) or disk arrays), and bridges and multiplexers.
  • Road multiplexer in which all devices are connected to Fiber Channel switches, is suitable for services requiring high throughput and low latency.
  • cloud storage service providers often build SANs inside data centers and implement security protection and data protection for the data centers internally. In this way, when the network quality of the data center is poor or the remaining capacity is small, The client's business requests, such as business I/O requests, cannot be processed in a timely manner, resulting in low efficiency in business request processing and the inability to ensure business continuity.
  • Embodiments of the present application provide a cross-regional data scheduling method, device, equipment and storage medium to improve service request processing efficiency and ensure business continuity.
  • embodiments of the present application provide a cross-region data scheduling method, including:
  • the target information of each data center is obtained, where the storage architecture of each data center is built based on the storage area network, and the target information includes at least one of metadata information and status information, so
  • the metadata information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the operating status description information of any data center;
  • the target data center Based on the data description information and each target information, determine the target data center corresponding to the service request, wherein the target data center is at least one of the various data centers;
  • the data plane interface information of the target data center is returned to the client, so that the client constructs an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  • the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
  • Determining the target data center corresponding to the service request based on the data description information and each target information includes:
  • the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
  • the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
  • the method further includes:
  • the method includes:
  • the existing metadata information is updated based on the first metadata information of the first data center and the second metadata information of the second data center.
  • the status information includes some or all of the following information:
  • the network device operating status description information of the storage area network
  • the metadata information includes some or all of the following information:
  • embodiments of the present application provide a cross-region data scheduling device, including:
  • a receiving module configured to receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
  • the acquisition module is used to obtain the target information of each data center based on the request operation type.
  • the storage architecture of each data center is built based on the storage area network.
  • the target information includes metadata information and status information. At least one, the metadata information represents the inventory data description information of each logical disk unit included in any data center, and the status information represents the operating status description information of any data center;
  • Determining module configured to determine the target data center corresponding to the service request based on the data description information and each target information, wherein the target data center is at least one of the various data centers;
  • a sending module configured to return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  • the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
  • the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
  • the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
  • the sending module after returning the data plane interface information of the target data center to the client, the sending module is also used to:
  • the determining module is used to migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the respective data centers, The second data center is any one of the data centers except the first data center; it is also used to, after completing the data migration, based on the first metadata of the first data center information, and the second metadata information of the second data center updates the existing metadata information.
  • the status information includes some or all of the following information:
  • the network device operating status description information of the storage area network
  • the metadata information includes some or all of the following information:
  • embodiments of the present application provide an electronic device, which includes a processor and a memory,
  • the memory is used to store computer programs or instructions
  • the processor is configured to execute computer programs or instructions in the memory, so that the method described in any one of the above first aspects is executed.
  • embodiments of the present application provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.
  • the target information of each data center is obtained, and then, based on the data description information of the data to be transmitted contained in the business request, the corresponding data center of the business request is determined.
  • the target data center and returns the data plane interface information of the target data center to the client, so that the client can build uplink and downlink data transmission channels corresponding to the business request based on the data plane interface information.
  • the storage architecture of each data center is based on The storage area network is built, and the target information includes at least one of metadata information and status information.
  • the metadata information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the storage data description information of any data center.
  • Running status description information the target data center is at least one of each data center; in this way, the client can build upstream and downstream channels corresponding to business requests with multiple data centers, unbinding the client from the default data center, and improving It improves the performance of storage services of the established storage area network architecture, thereby improving the efficiency of business request processing and ensuring business continuity to the greatest extent.
  • Figure 1 is a schematic diagram of an application scenario in an embodiment of the present application
  • Figure 2 is a schematic architectural diagram of a cross-region data scheduling system in an embodiment of the present application
  • Figure 3 is a schematic flowchart of a cross-region data scheduling method in an embodiment of the present application.
  • Figure 4 is a schematic diagram of the processing logic of the cross-region data scheduling system in the embodiment of the present application.
  • Figure 5 is a schematic flowchart of a method for determining a target data center in an embodiment of the present application
  • Figure 6 is a schematic flowchart of another method for determining a target data center in an embodiment of the present application.
  • Figure 7 is a logical schematic diagram of the cross-region data scheduling system performing data scheduling through the scheduling module in the embodiment of the present application;
  • Figure 8 is a schematic flowchart of a method for updating existing metadata information in target information in an embodiment of the present application
  • Figure 9 is a schematic flowchart of another cross-region data scheduling method in an embodiment of the present application.
  • Figure 10 is a schematic diagram of the logical architecture of a cross-region data scheduling device in an embodiment of the present application.
  • Figure 11 is a schematic diagram of the physical architecture of an electronic device in an embodiment of the present application.
  • the target information of each data center is obtained based on the request operation type contained in the received service request sent by the client. , and then, based on the data description information of the data to be transmitted contained in the business request, determine the target data center corresponding to the business request, and return the data plane interface information of the target data center to the client, so that the client can build a data center based on the data plane interface information.
  • the storage architecture of each data center is based on the storage area network.
  • the target information includes at least one of metadata information and status information.
  • the metadata information represents any data center. It contains the inventory data description information of each logical disk unit, and the status information represents the operating status description information of any data center.
  • the target data center is at least one of each data center.
  • the client can build uplink and downlink channels corresponding to business requests with multiple data centers, unbinding the client from the default data center, improving the performance of storage services that have built a storage area network architecture, thereby improving business Request processing efficiency ensures business continuity to the greatest extent.
  • the above method realizes load balancing scheduling for multiple data centers based on status information, making incremental data and existing data schedulable for migration, realizing imperceptible expansion and migration, and also enhancing the overall availability of the storage area. Expandability.
  • Figure 1 shows a schematic diagram of an application scenario according to the embodiment of the present application.
  • the above application scenario includes a client 1, a cross-region data scheduling system 2 and a data center 3, where the client 1 and the cross-region data scheduling system 2 are connected through a communication network.
  • the communication network may be a wired network or a wireless network.
  • the client 1 and the cross-region data scheduling system 2 can be connected directly or indirectly through wired or wireless communication methods, which is not limited in this application.
  • the cross-regional data scheduling system 2 and the data center 3 can also communicate through a communication network, and the communication network can also be a wired network or a wireless network.
  • the cross-regional data scheduling system 2 and the data center 3 can be connected directly or indirectly through wired or wireless communication methods, and this application is not limited here.
  • the cross-region data scheduling system 2 is an electronic device equipped with the program of the cross-regional data scheduling method provided by the embodiment of the present application.
  • the electronic device may be a smart terminal, a computer, a tablet, a notebook, or an e-book. Readers and other devices; it can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, Cloud communications, middleware services, domain name services, security services, content distribution network (Content Delivery Network (CDN), as well as cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.
  • CDN Content Distribution Network
  • Figure 2 shows a schematic architectural diagram of a cross-region data scheduling system provided in this embodiment of the present application.
  • the cross-region data scheduling system includes a scheduling module, a collection module and a metadata server, where,
  • the metadata server is used to record the metadata information of each logical disk unit contained in each managed data center, and is also used to record the corresponding status information of each data center; among which, the metadata information and/or status information is used to
  • the scheduling module performs data scheduling on business requests between data plane interfaces in different data centers.
  • the collection module is used to regularly obtain the status information of different data centers through the respective status information interfaces of each data center, and record or update the status information to the metadata server.
  • the scheduling module is used to collect the metadata information of the existing data in each data center and record it in the metadata server; it is also used to record the data plane interface information of each managed data center.
  • Data information or each status information is used to make decisions, determine the data plane interface information of the target data center for real-time business requests, and migrate and schedule the existing data based on the status information of each data center; it is also used for business requests (such as writing After the incoming request) is completed, the metadata information recorded in the metadata server is updated based on the written target metadata information of the target data center.
  • collection module, metadata server and scheduling module included in the cross-region data scheduling system provided by this application can be deployed in one physical device at the same time, or can be deployed in different physical devices respectively, which will not be discussed here. Specific limitations.
  • an embodiment of the present application provides a cross-region data scheduling method.
  • the specific process is as follows:
  • Step 300 Receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted.
  • step 300 before executing step 300, it is necessary to construct respective status information interfaces in each data center included in the cross-regional data scheduling system, which is used to regularly obtain the status of each data center through the collection module. information, and record or update the obtained status information to the metadata server.
  • the cross-regional data scheduling system collects the data plane interface information of each data center through the scheduling module (the data plane interface is an existing interface, this application can directly collect the data plane interface information for use), and based on each data plane interface Information, collect metadata information of the existing data in each data center, and record the collected metadata information into the metadata server, thereby obtaining the initial metadata information of each data center.
  • step 300 is executed to receive the service request sent by the client.
  • the service requests involved in the embodiments of this application include service I/O requests. Therefore, each service request includes the requested operation type and data description information of the data to be transmitted, where the data description information includes data offset and data length.
  • the data to be transmitted may be data to be read from one or some data centers, data to be written to one or some data centers, or the above-mentioned data. Data to be read and data to be written.
  • Step 310 Based on the requested operation type, obtain the target information of each data center.
  • the storage architecture of each data center is built based on the storage area network.
  • the target information includes at least one of metadata information and status information.
  • Metadata The information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the operating status description information of any data center.
  • step 310 when step 310 is executed, the target information of each data center is obtained based on the request operation type included in the received business request. That is, if the request operation type is a read operation, the metadata of each data center is obtained. Information, if the requested operation type is a write operation, obtain the status information of each data center.
  • Metadata information includes some or all of the following information:
  • status information includes some or all of the following information:
  • Step 320 Determine the target data center corresponding to the service request based on the data description information and each target information, where the target data center is at least one of each data center.
  • step 320 when step 320 is executed, different data scheduling processes are executed based on different request operation types.
  • Step 3201 Query the existing data corresponding to the data description information from each metadata information.
  • Step 3202 Determine the data center corresponding to the queried data center index information of the existing data as the target data center.
  • Step 3201' Determine the scheduling plan corresponding to the data to be transmitted based on the data description information and each status information.
  • the status information is regularly collected through the status information interface corresponding to any data center.
  • Step 3202' Based on the scheduling plan, determine the target data center.
  • Step 330 Return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  • step 320 after executing step 320 and determining the target data center corresponding to the service request, since the cross-regional data scheduling system has collected the data plane interface information of each data center through the scheduling module, then when executing step 330, Return the data plane interface information of the target data center to the client. In this way, the client can build uplink and downlink data transmission channels with the target data center based on the data plane interface information returned by the cross-regional data scheduling system, thereby realizing the data to be transmitted. data transmission.
  • each metadata information is obtained from the metadata server through the scheduling module, and a search operation is performed, that is, the existing data corresponding to the data description information is queried from each metadata information, and The data center corresponding to the data center index information of the queried stock data is determined as the target data center;
  • the requested operation type is a write operation
  • the status information is obtained from the metadata server through the scheduling module and the decision-making operation is performed, that is Based on the data description information and each status information, the scheduling plan corresponding to the data to be transmitted is determined; and based on the scheduling plan, the target data center is determined.
  • the data plane interface information of the target data center is returned to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information, so as to facilitate subsequent smooth transmission of the data to be transmitted.
  • step 330 if the request operation type included in the service request is a write operation, refer to Figure 8 to further perform the following steps:
  • Step 800 After determining that the write operation is completed, collect target metadata information of the target data center.
  • step 800 when step 800 is executed, after it is determined that the write operation is completed, the target metadata information of the target data center is collected through the scheduling module.
  • Step 810 Based on the target metadata information, update the existing metadata information in the target information.
  • step 810 when step 810 is executed, based on the target metadata information, the existing metadata information in the target information that has been recorded (that is, stored) in the metadata server is updated. In this way, it can be ensured that the The stored metadata information is consistent with the metadata information corresponding to each logical disk unit in each data center, which facilitates the subsequent accurate and rapid execution of newly received business requests, thereby ensuring business continuity.
  • the cross-regional data scheduling system can use the scheduling module to perform operations on the existing data in one or some data centers.
  • data migration refer to Figure 9, taking data migration between the first data center and the second data center as an example.
  • the specific data scheduling process includes:
  • Step 900 Migrate the target stock data of the first data center to the second data center, where the first data center is any one of the data centers, and the second data center is any one of the data centers, except the first data center. any data center outside.
  • Step 910 After completing the data migration, update the existing metadata information based on the first metadata information of the first data center and the second metadata information of the second data center.
  • steps 900 to 910 only take data migration between the first data center and the second data center as an example.
  • the data migration may be performed between the first data center, the second data center, and the third data center.
  • Data migration between centers or even between more data centers is not specifically limited in this application.
  • part of the target inventory data in the first data center is migrated to the second data center, the remaining target inventory data is migrated to the third data center, and so on.
  • the cross-region data scheduling method adopted in the embodiment of this application improves the performance of storage services of the established storage area network architecture through the above-mentioned cross-region data scheduling system; several data centers are managed through the scheduling module to facilitate customers It builds uplink and downlink data transmission channels for business requests with multiple data centers, thereby realizing load balancing scheduling based on status information; the above cross-regional data scheduling system also enhances the overall scalability of storage services, and can increase or decrease storage capacity by increasing or decreasing capacity.
  • the number of managed data centers enables parallel expansion and contraction.
  • the data scheduling and data migration of incremental data and stock data through the scheduling module realizes imperceptible expansion and migration.
  • the internal storage resources of the managed data center are recycled based on metadata information, improving The infrastructure utilization of each data center.
  • an embodiment of the present application provides a cross-region data scheduling device, including:
  • the receiving module 1010 is used to receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
  • the acquisition module 1020 is used to obtain the target information of each data center based on the requested operation type.
  • the storage architecture of each data center is built based on the storage area network.
  • the target information includes metadata information and status information. At least one of, the metadata information represents the inventory data description information of each logical disk unit included in any data center, and the status information represents the operating status description information of any data center;
  • Determining module 1030 configured to determine the target data center corresponding to the service request based on the data description information and each target information, wherein the target data center is at least one of the various data centers;
  • the sending module 1040 is configured to return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  • the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
  • the determination module 1030 Determining the target data center corresponding to the service request based on the data description information and each target information, the determination module 1030 is used to:
  • the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
  • the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
  • the sending module 1040 is also used to:
  • the determining module 1030 is used to migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the respective data centers.
  • the second data center is any one of the data centers except the first data center; it is also used to, after the data migration is completed, the first element based on the first data center
  • the data information, and the second metadata information of the second data center update the existing metadata information.
  • the status information includes some or all of the following information:
  • the network device operating status description information of the storage area network
  • the metadata information includes some or all of the following information:
  • an electronic device including: a processor 1101 and a memory 1102;
  • Memory 1102 is used to store computer programs executed by the processor 1101.
  • the memory 1102 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory (non-volatile memory).
  • memory such as read-only memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD), or the memory 1102 is capable of carrying or storing instructions or data.
  • any other medium that may be in the form of the desired program code and capable of being accessed by a computer.
  • the memory 1102 may be a combination of the above-described memories.
  • Processor 1101 may include one or more central processing units (central processing units). processing unit (CPU), graphics processing unit (GPU) or digital processing unit, etc.
  • CPU central processing unit
  • GPU graphics processing unit
  • digital processing unit etc.
  • the specific connection medium between the above-mentioned memory 1102 and the processor 1101 is not limited in the embodiment of the present application.
  • the memory 1102 and the processor 1101 are connected through a bus 1103 in Figure 11.
  • the bus 1103 is represented by a thick line in Figure 11.
  • the bus 1103 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.
  • the memory stores program code.
  • the program code When executed by the processor 1101, it causes the processor 1101 to perform any of the methods performed in the above embodiments.
  • the electronic device is an electronic device that performs the method in the embodiment of the present application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation of the method, and repeated details will not be repeated.
  • embodiments of the present application provide a computer-readable storage medium on which computer program instructions are stored.
  • the computer program instructions are executed by a processor, any one of the methods in the above embodiments can be implemented.
  • embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions
  • the device implements the functions specified in the process or processes in the flowchart and/or the block or blocks in the block diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device.
  • Instructions provide steps for implementing the functions specified in a process or processes in a flowchart diagram and/or in a block or blocks in a block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present application relates to the technical field of cloud computing, and in particular, to a cross-region data scheduling method and apparatus, a device, and a storage medium, which are used for solving the problems in the prior art that the service request processing efficiency is low and the service continuity cannot be ensured. The method comprises: acquiring target information of data centers on the basis of a request operation type comprised in a service request sent by a client, and on the basis of data description information of data to be transmitted comprised in the service request and the target information, determining and sending data plane interface information of a target data center corresponding to the service request to the client, wherein the target information comprises at least one of metadata information and state information, the metadata information represents base data description information of each logical disk unit comprised in any data center, and the state information represents operation state description information of any data center. In this way, the service request processing efficiency is improved, and the service continuity is ensured.

Description

一种跨区域的数据调度方法、装置、设备及存储介质A cross-regional data scheduling method, device, equipment and storage medium 技术领域Technical field
本申请涉及云计算技术领域,尤其涉及一种跨区域的数据调度方法、装置、设备及存储介质。The present application relates to the field of cloud computing technology, and in particular to a cross-regional data scheduling method, device, equipment and storage medium.
背景技术Background technique
随着云计算技术的发展,具有易用易接入、无需客户构建数据中心等的云存储越来越受到大家的认可。存储区域网络(Storage Area Network,SAN)是一种以网络为中心的存储结构,通常包括服务器(主机)、存储设备(逻辑磁盘单元(Logical Disk Unit,LUN)或磁盘阵列)以及桥接器和多路复用器,其中,所有设备都连接在光纤通道的交换机上,适用于要求高吞吐量和低延迟的业务。With the development of cloud computing technology, cloud storage that is easy to use and access and does not require customers to build a data center is increasingly recognized by everyone. Storage Area Network (SAN) is a network-centric storage structure that usually includes servers (hosts), storage devices (logical disk units (Logical Disk Units, LUNs) or disk arrays), and bridges and multiplexers. Road multiplexer, in which all devices are connected to Fiber Channel switches, is suitable for services requiring high throughput and low latency.
技术问题technical problem
现有技术下,云存储服务商往往在数据中心内部构建SAN,在内部实施对该数据中心的安全防护和数据保护等,这样,当该数据中心的网络质量较差或容量剩余较少时,不能及时处理客户端的业务请求,如业务I/O请求,从而造成业务请求处理效率低下,无法保证业务的连续性的问题。Under the existing technology, cloud storage service providers often build SANs inside data centers and implement security protection and data protection for the data centers internally. In this way, when the network quality of the data center is poor or the remaining capacity is small, The client's business requests, such as business I/O requests, cannot be processed in a timely manner, resulting in low efficiency in business request processing and the inability to ensure business continuity.
技术解决方案Technical solutions
本申请实施例提供一种跨区域的数据调度方法、装置、设备及存储介质,用以提高业务请求处理效率,保证业务的连续性。Embodiments of the present application provide a cross-regional data scheduling method, device, equipment and storage medium to improve service request processing efficiency and ensure business continuity.
本申请实施例提供的具体技术方案如下:The specific technical solutions provided by the embodiments of this application are as follows:
第一方面,本申请实施例提供一种跨区域的数据调度方法,包括:In the first aspect, embodiments of the present application provide a cross-region data scheduling method, including:
接收客户端发送的业务请求,其中,所述业务请求包括请求操作类型、待传输数据的数据描述信息;Receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
基于所述请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,所述目标信息包括元数据信息和状态信息中的至少一种,所述元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,所述状态信息表征任一数据中心的运行状态描述信息;Based on the request operation type, the target information of each data center is obtained, where the storage architecture of each data center is built based on the storage area network, and the target information includes at least one of metadata information and status information, so The metadata information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the operating status description information of any data center;
基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,其中,所述目标数据中心是所述各个数据中心中的至少一个;Based on the data description information and each target information, determine the target data center corresponding to the service request, wherein the target data center is at least one of the various data centers;
将所述目标数据中心的数据面接口信息返回至所述客户端,以使所述客户端基于所述数据面接口信息构建所述业务请求对应的上下行数据传输通道。The data plane interface information of the target data center is returned to the client, so that the client constructs an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
在一些可能的实施例中,若所述请求操作类型是读操作,则所述目标信息是所述元数据信息;若所述请求操作类型是写操作,则所述目标信息是所述状态信息;In some possible embodiments, if the requested operation type is a read operation, the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
所述基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,包括:Determining the target data center corresponding to the service request based on the data description information and each target information includes:
若所述请求操作类型是读操作,则从所述各个元数据信息中,查询所述数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为所述目标数据中心;If the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
若所述请求操作类型是写操作,则基于所述数据描述信息和所述各个状态信息,确定所述待传输数据对应的调度计划,并基于所述调度计划,确定所述目标数据中心,其中,所述状态信息是通过所述任一数据中心对应的状态信息接口定期采集得到的。If the requested operation type is a write operation, determine the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
在一些可能的实施例中,在所述将所述目标数据中心的数据面接口信息返回至所述客户端之后,还包括:In some possible embodiments, after returning the data plane interface information of the target data center to the client, the method further includes:
在确定所述写操作完成后,收集所述目标数据中心的目标元数据信息;After determining that the write operation is completed, collect target metadata information of the target data center;
基于所述目标元数据信息,更新所述目标信息中的已有元数据信息。Based on the target metadata information, existing metadata information in the target information is updated.
在一些可能的实施例中,所述方法包括:In some possible embodiments, the method includes:
将第一数据中心的目标存量数据迁移到第二数据中心中,其中,所述第一数据中心是所述各个数据中心中的任意一个,所述第二数据中心是所述各个数据中心中,除所述第一数据中心之外的任意一个数据中心;Migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the data centers, and the second data center is one of the data centers, Any data center except the first data center;
在完成数据迁移后,基于所述第一数据中心的第一元数据信息,以及,所述第二数据中心的第二元数据信息,更新已有的元数据信息。After the data migration is completed, the existing metadata information is updated based on the first metadata information of the first data center and the second metadata information of the second data center.
在一些可能的实施例中,所述状态信息包括如下信息中的部分或全部:In some possible embodiments, the status information includes some or all of the following information:
网络运行状态描述信息;Network operating status description information;
所述存储区域网络的网络设备运行状态描述信息;The network device operating status description information of the storage area network;
辅助设备运行状态描述信息。Auxiliary equipment operating status description information.
在一些可能的实施例中,所述元数据信息包括如下信息中的部分或全部:In some possible embodiments, the metadata information includes some or all of the following information:
数据中心索引信息;Data center index information;
每个逻辑磁盘单元的存量数据的数据偏移;Data offset of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的数据长度;The data length of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的时间戳。The timestamp of the inventory data for each logical disk unit.
第二方面,本申请实施例提供一种跨区域的数据调度装置,包括:In the second aspect, embodiments of the present application provide a cross-region data scheduling device, including:
接收模块,用于接收客户端发送的业务请求,其中,所述业务请求包括请求操作类型、待传输数据的数据描述信息;A receiving module, configured to receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
获取模块,用于基于所述请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,所述目标信息包括元数据信息和状态信息中的至少一种,所述元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,所述状态信息表征任一数据中心的运行状态描述信息;The acquisition module is used to obtain the target information of each data center based on the request operation type. The storage architecture of each data center is built based on the storage area network. The target information includes metadata information and status information. At least one, the metadata information represents the inventory data description information of each logical disk unit included in any data center, and the status information represents the operating status description information of any data center;
确定模块,用于基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,其中,所述目标数据中心是所述各个数据中心中的至少一个;Determining module, configured to determine the target data center corresponding to the service request based on the data description information and each target information, wherein the target data center is at least one of the various data centers;
发送模块,用于将所述目标数据中心的数据面接口信息返回至所述客户端,以使所述客户端基于所述数据面接口信息构建所述业务请求对应的上下行数据传输通道。A sending module, configured to return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
在一些可能的实施例中,若所述请求操作类型是读操作,则所述目标信息是所述元数据信息;若所述请求操作类型是写操作,则所述目标信息是所述状态信息;In some possible embodiments, if the requested operation type is a read operation, the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
所述基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,所述确定模块用于:Determine the target data center corresponding to the service request based on the data description information and each target information, and the determination module is used to:
若所述请求操作类型是读操作,则从所述各个元数据信息中,查询所述数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为所述目标数据中心;If the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
若所述请求操作类型是写操作,则基于所述数据描述信息和所述各个状态信息,确定所述待传输数据对应的调度计划,并基于所述调度计划,确定所述目标数据中心,其中,所述状态信息是通过所述任一数据中心对应的状态信息接口定期采集得到的。If the requested operation type is a write operation, determine the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
在一些可能的实施例中,在所述将所述目标数据中心的数据面接口信息返回至所述客户端之后,所述发送模块还用于:In some possible embodiments, after returning the data plane interface information of the target data center to the client, the sending module is also used to:
在确定所述写操作完成后,收集所述目标数据中心的目标元数据信息;After determining that the write operation is completed, collect target metadata information of the target data center;
基于所述目标元数据信息,更新所述目标信息中的已有元数据信息。Based on the target metadata information, existing metadata information in the target information is updated.
在一些可能的实施例中,所述确定模块用于将第一数据中心的目标存量数据迁移到第二数据中心中,其中,所述第一数据中心是所述各个数据中心中的任意一个,所述第二数据中心是所述各个数据中心中,除所述第一数据中心之外的任意一个数据中心;还用于在完成数据迁移后,基于所述第一数据中心的第一元数据信息,以及,所述第二数据中心的第二元数据信息,更新已有的元数据信息。In some possible embodiments, the determining module is used to migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the respective data centers, The second data center is any one of the data centers except the first data center; it is also used to, after completing the data migration, based on the first metadata of the first data center information, and the second metadata information of the second data center updates the existing metadata information.
在一些可能的实施例中,所述状态信息包括如下信息中的部分或全部:In some possible embodiments, the status information includes some or all of the following information:
网络运行状态描述信息;Network operating status description information;
所述存储区域网络的网络设备运行状态描述信息;The network device operating status description information of the storage area network;
辅助设备运行状态描述信息。Auxiliary equipment operating status description information.
在一些可能的实施例中,所述元数据信息包括如下信息中的部分或全部:In some possible embodiments, the metadata information includes some or all of the following information:
数据中心索引信息;Data center index information;
每个逻辑磁盘单元的存量数据的数据偏移;Data offset of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的数据长度;The data length of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的时间戳。The timestamp of the inventory data for each logical disk unit.
第三方面,本申请实施例提供一种电子设备,所述电子设备包括处理器和存储器,In a third aspect, embodiments of the present application provide an electronic device, which includes a processor and a memory,
所述存储器,用于存储计算机程序或指令;The memory is used to store computer programs or instructions;
所述处理器,用于执行存储器中的计算机程序或指令,使得上述第一方面中任一项所述的方法被执行。The processor is configured to execute computer programs or instructions in the memory, so that the method described in any one of the above first aspects is executed.
第四方面,本申请实施例提供一种计算机可读存储介质,其上存储有计算机程序指令,该计算机程序指令被处理器执行时实现上述第一方面中任一项所述方法的步骤。In a fourth aspect, embodiments of the present application provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the steps of any one of the methods described in the first aspect are implemented.
本申请实施例中,基于接收到的客户端发送的业务请求包含的请求操作类型,获取各个数据中心的目标信息,然后,基于业务请求包含的待传输数据的数据描述信息,确定业务请求对应的目标数据中心,并将目标数据中心的数据面接口信息返回至客户端,以使客户端基于数据面接口信息构建业务请求对应的上下行数据传输通道,其中,每个数据中心的存储架构是基于存储区域网络搭建的,目标信息包括元数据信息和状态信息中的至少一种,元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,状态信息表征任一数据中心的运行状态描述信息,目标数据中心是各个数据中心中的至少一个;这样,客户端可以与多个数据中心构建业务请求对应的上下行通道,解除了客户端与默认数据中心的绑定关系,提升了已搭建存储区域网络架构的存储服务的性能,从而提高了业务请求处理效率,最大限度保证了业务的连续性。In the embodiment of this application, based on the request operation type contained in the received business request sent by the client, the target information of each data center is obtained, and then, based on the data description information of the data to be transmitted contained in the business request, the corresponding data center of the business request is determined. The target data center, and returns the data plane interface information of the target data center to the client, so that the client can build uplink and downlink data transmission channels corresponding to the business request based on the data plane interface information. The storage architecture of each data center is based on The storage area network is built, and the target information includes at least one of metadata information and status information. The metadata information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the storage data description information of any data center. Running status description information, the target data center is at least one of each data center; in this way, the client can build upstream and downstream channels corresponding to business requests with multiple data centers, unbinding the client from the default data center, and improving It improves the performance of storage services of the established storage area network architecture, thereby improving the efficiency of business request processing and ensuring business continuity to the greatest extent.
附图说明Description of drawings
图1为本申请实施例中一种应用场景的示意图;Figure 1 is a schematic diagram of an application scenario in an embodiment of the present application;
图2为本申请实施例中一种跨区域数据调度系统的架构示意图;Figure 2 is a schematic architectural diagram of a cross-region data scheduling system in an embodiment of the present application;
图3为本申请实施例中一种跨区域的数据调度方法的流程示意图;Figure 3 is a schematic flowchart of a cross-region data scheduling method in an embodiment of the present application;
图4为本申请实施例中跨区域数据调度系统的处理逻辑示意图;Figure 4 is a schematic diagram of the processing logic of the cross-region data scheduling system in the embodiment of the present application;
图5为本申请实施例中一种确定目标数据中心方法的流程示意图;Figure 5 is a schematic flowchart of a method for determining a target data center in an embodiment of the present application;
图6为本申请实施例中另一种确定目标数据中心方法的流程示意图;Figure 6 is a schematic flowchart of another method for determining a target data center in an embodiment of the present application;
图7为本申请实施例中跨区域数据调度系统通过调度模块进行数据调度的逻辑示意图;Figure 7 is a logical schematic diagram of the cross-region data scheduling system performing data scheduling through the scheduling module in the embodiment of the present application;
图8为本申请实施例中一种更新目标信息中的已有元数据信息方法的流程示意图;Figure 8 is a schematic flowchart of a method for updating existing metadata information in target information in an embodiment of the present application;
图9为本申请实施例中另一种跨区域的数据调度方法的流程示意图;Figure 9 is a schematic flowchart of another cross-region data scheduling method in an embodiment of the present application;
图10为本申请实施例中一种跨区域的数据调度装置的逻辑架构示意图;Figure 10 is a schematic diagram of the logical architecture of a cross-region data scheduling device in an embodiment of the present application;
图11为本申请实施例中电子设备的实体架构示意图。Figure 11 is a schematic diagram of the physical architecture of an electronic device in an embodiment of the present application.
本发明的实施方式Embodiments of the invention
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,并不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.
需要说明的是,本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、“第四”等是用于区别类似的对象,而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换,以便这里描述的本申请的实施例能够在除了这里图示或描述的那些以外的顺序实施。It should be noted that the terms "first", "second", "third", "fourth", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily Used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application described herein are capable of being practiced in sequences other than those illustrated or described herein.
为了解决已有技术下业务请求处理效率低下,无法保证业务的连续性的问题,本申请实施例中,基于接收到的客户端发送的业务请求包含的请求操作类型,获取各个数据中心的目标信息,然后,基于业务请求包含的待传输数据的数据描述信息,确定业务请求对应的目标数据中心,并将目标数据中心的数据面接口信息返回至客户端,以使客户端基于数据面接口信息构建业务请求对应的上下行数据传输通道,其中,每个数据中心的存储架构是基于存储区域网络搭建的,目标信息包括元数据信息和状态信息中的至少一种,元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,状态信息表征任一数据中心的运行状态描述信息,目标数据中心是各个数据中心中的至少一个。In order to solve the problem of low service request processing efficiency and inability to ensure service continuity under the existing technology, in the embodiment of this application, the target information of each data center is obtained based on the request operation type contained in the received service request sent by the client. , and then, based on the data description information of the data to be transmitted contained in the business request, determine the target data center corresponding to the business request, and return the data plane interface information of the target data center to the client, so that the client can build a data center based on the data plane interface information. Uplink and downlink data transmission channels corresponding to business requests. The storage architecture of each data center is based on the storage area network. The target information includes at least one of metadata information and status information. The metadata information represents any data center. It contains the inventory data description information of each logical disk unit, and the status information represents the operating status description information of any data center. The target data center is at least one of each data center.
这样,客户端可以与多个数据中心构建业务请求对应的上下行通道,解除了客户端与默认数据中心的绑定关系,提升了已搭建存储区域网络架构的存储服务的性能,从而提高了业务请求处理效率,最大限度保证了业务的连续性。In this way, the client can build uplink and downlink channels corresponding to business requests with multiple data centers, unbinding the client from the default data center, improving the performance of storage services that have built a storage area network architecture, thereby improving business Request processing efficiency ensures business continuity to the greatest extent.
进一步地,上述方法,根据状态信息实现了对多个数据中心进行的负载均衡调度,使得增量数据和存量数据可调度迁移,实现了无感知的扩容与迁移,也增强了存储区域整体的可拓展性。Furthermore, the above method realizes load balancing scheduling for multiple data centers based on status information, making incremental data and existing data schedulable for migration, realizing imperceptible expansion and migration, and also enhancing the overall availability of the storage area. Expandability.
下面结合附图对本申请优选的实施方式做出进一步详细说明,应当理解,此处所描述的优选实施例仅用于说明和解释本申请,并不用于限定本申请,并且在不冲突的情况下,本申请实施例及实施例中的特征可以相互组合。The preferred embodiments of the present application will be described in further detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described here are only used to illustrate and explain the present application and are not used to limit the present application. In the absence of conflict, The embodiments of the present application and the features in the embodiments can be combined with each other.
图1示出了本申请实施例的一种应用场景的示意图。参阅图1所示,本申请实施例中,上述应用场景中包括客户端1、跨区域数据调度系统2和数据中心3,其中,客户端1与跨区域数据调度系统2之间通过通信网络进行通信。可选地,通信网络可以是有线网络或无线网络。客户端1以及跨区域数据调度系统2可以通过有线或无线通信方式进行直接或间接地连接,本申请在此不做限制。Figure 1 shows a schematic diagram of an application scenario according to the embodiment of the present application. Referring to Figure 1 , in the embodiment of the present application, the above application scenario includes a client 1, a cross-region data scheduling system 2 and a data center 3, where the client 1 and the cross-region data scheduling system 2 are connected through a communication network. communication. Alternatively, the communication network may be a wired network or a wireless network. The client 1 and the cross-region data scheduling system 2 can be connected directly or indirectly through wired or wireless communication methods, which is not limited in this application.
相应的,跨区域数据调度系统2与数据中心3之间也可以通过通信网络进行通信,通信网络也可以是有线网络或无线网络。跨区域数据调度系统2和数据中心3可以通过有线或无线通信方式进行直接或间接地连接,本申请在此也不做限制。Correspondingly, the cross-regional data scheduling system 2 and the data center 3 can also communicate through a communication network, and the communication network can also be a wired network or a wireless network. The cross-regional data scheduling system 2 and the data center 3 can be connected directly or indirectly through wired or wireless communication methods, and this application is not limited here.
在本申请实施例中,跨区域数据调度系统2为搭载本申请实施例提供的跨区域的数据调度方法的程序的电子设备,该电子设备可以是智能终端、计算机、平板电脑、笔记本、电子书阅读器等设备;也可以是独立的物理服务器,可以是多个物理服务器构成的服务器集群或者分布式系统,还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(Content Delivery Network,CDN)、以及大数据和人工智能平台等基础云计算服务的云服务器。In the embodiment of the present application, the cross-region data scheduling system 2 is an electronic device equipped with the program of the cross-regional data scheduling method provided by the embodiment of the present application. The electronic device may be a smart terminal, a computer, a tablet, a notebook, or an e-book. Readers and other devices; it can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or it can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, Cloud communications, middleware services, domain name services, security services, content distribution network (Content Delivery Network (CDN), as well as cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.
图2示出了本申请实施例中提供的一种跨区域数据调度系统的架构示意图。参阅图2所示,本申请实施例中,该跨区域数据调度系统包括调度模块、采集模块和元数据服务器,其中,Figure 2 shows a schematic architectural diagram of a cross-region data scheduling system provided in this embodiment of the present application. Referring to Figure 2, in this embodiment of the present application, the cross-region data scheduling system includes a scheduling module, a collection module and a metadata server, where,
元数据服务器,用于记录纳管的各个数据中心包含的每个逻辑磁盘单元的元数据信息,还用于记录各个数据中心各自对应的状态信息;其中,元数据信息和/或状态信息用于通过调度模块对业务请求在不同数据中心的数据面接口间进行数据调度。The metadata server is used to record the metadata information of each logical disk unit contained in each managed data center, and is also used to record the corresponding status information of each data center; among which, the metadata information and/or status information is used to The scheduling module performs data scheduling on business requests between data plane interfaces in different data centers.
采集模块,用于通过各个数据中心各自的状态信息接口,定期获取不同的数据中心的状态信息,并记录或更新该状态信息到元数据服务器中。The collection module is used to regularly obtain the status information of different data centers through the respective status information interfaces of each data center, and record or update the status information to the metadata server.
调度模块,用于收集各个数据中心的存量数据的元数据信息,将其记录到元数据服务器中;还用于记录纳管的各个数据中心的数据面接口信息,根据元数据服务器中的各个元数据信息或各个状态信息进行决策,为实时的业务请求确定目标数据中心的数据面接口信息,以及根据各个数据中心的状态信息,对存量数据进行迁移调度;也用于在业务请求(如,写入请求)完成后,基于写入后的目标数据中心的目标元数据信息,更新元数据服务器中记录的元数据信息。The scheduling module is used to collect the metadata information of the existing data in each data center and record it in the metadata server; it is also used to record the data plane interface information of each managed data center. According to each element in the metadata server Data information or each status information is used to make decisions, determine the data plane interface information of the target data center for real-time business requests, and migrate and schedule the existing data based on the status information of each data center; it is also used for business requests (such as writing After the incoming request) is completed, the metadata information recorded in the metadata server is updated based on the written target metadata information of the target data center.
需要说明的是,本申请提供的跨区域数据调度系统包含的采集模块、元数据服务器和调度模块,可以同时部署在一个实体设备中,也可以分别部署在不同的实体设备中,在此不做具体限定。It should be noted that the collection module, metadata server and scheduling module included in the cross-region data scheduling system provided by this application can be deployed in one physical device at the same time, or can be deployed in different physical devices respectively, which will not be discussed here. Specific limitations.
参阅图3所示,本申请实施例提供的一种跨区域的数据调度方法,具体流程如下:Referring to Figure 3, an embodiment of the present application provides a cross-region data scheduling method. The specific process is as follows:
步骤300:接收客户端发送的业务请求,其中,业务请求包括请求操作类型、待传输数据的数据描述信息。Step 300: Receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted.
参阅图4所示,本申请实施例中,在执行步骤300之前,需在纳入跨区域数据调度系统的各个数据中心中构建各自的状态信息接口,用于通过采集模块定期获取各个数据中心的状态信息,并将获取到的状态信息记录或更新到元数据服务器中。Referring to Figure 4, in the embodiment of the present application, before executing step 300, it is necessary to construct respective status information interfaces in each data center included in the cross-regional data scheduling system, which is used to regularly obtain the status of each data center through the collection module. information, and record or update the obtained status information to the metadata server.
然后,跨区域数据调度系统通过调度模块收集各个数据中心的数据面接口信息(其中,数据面接口为现有接口,本申请直接收集数据面接口信息进行使用即可),并基于各个数据面接口信息,收集各个数据中心的存量数据的元数据信息,以及将收集到的元数据信息记录到元数据服务器中,从而得到各个数据中心的初始的元数据信息。Then, the cross-regional data scheduling system collects the data plane interface information of each data center through the scheduling module (the data plane interface is an existing interface, this application can directly collect the data plane interface information for use), and based on each data plane interface Information, collect metadata information of the existing data in each data center, and record the collected metadata information into the metadata server, thereby obtaining the initial metadata information of each data center.
在将跨区域数据调度系统的架构搭建完成后,执行步骤300,接收客户端发送的业务请求。本申请实施例所涉及的业务请求包括业务I/O请求,因此,每个业务请求包括请求操作类型、待传输数据的数据描述信息,其中,数据描述信息包括数据偏移和数据长度。After the architecture of the cross-region data scheduling system is completed, step 300 is executed to receive the service request sent by the client. The service requests involved in the embodiments of this application include service I/O requests. Therefore, each service request includes the requested operation type and data description information of the data to be transmitted, where the data description information includes data offset and data length.
一些实施例中,待传输数据可以是从某个或某些数据中心中读取的待读取数据,也可以是欲写入某个或某些数据中心的待写入数据,还可以是上述待读取数据和待写入数据。In some embodiments, the data to be transmitted may be data to be read from one or some data centers, data to be written to one or some data centers, or the above-mentioned data. Data to be read and data to be written.
步骤310:基于请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,目标信息包括元数据信息和状态信息中的至少一种,元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,状态信息表征任一数据中心的运行状态描述信息。Step 310: Based on the requested operation type, obtain the target information of each data center. The storage architecture of each data center is built based on the storage area network. The target information includes at least one of metadata information and status information. Metadata The information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the operating status description information of any data center.
本申请实施例中,在执行步骤310时,基于接收到的业务请求所包含的请求操作类型,获取各个数据中心的目标信息,即若请求操作类型是读操作,则获取各个数据中心的元数据信息,若请求操作类型是写操作,则获取各个数据中心的状态信息。In the embodiment of this application, when step 310 is executed, the target information of each data center is obtained based on the request operation type included in the received business request. That is, if the request operation type is a read operation, the metadata of each data center is obtained. Information, if the requested operation type is a write operation, obtain the status information of each data center.
本申请实施例中,元数据信息包括如下信息中的部分或全部:In the embodiment of this application, metadata information includes some or all of the following information:
1、数据中心索引信息;1. Data center index information;
2、每个逻辑磁盘单元的存量数据的数据偏移;2. Data offset of the existing data of each logical disk unit;
3、每个逻辑磁盘单元的存量数据的数据长度;3. The data length of the existing data of each logical disk unit;
4、每个逻辑磁盘单元的存量数据的时间戳。4. The timestamp of the existing data of each logical disk unit.
实际应用中,参阅表1所示,可以通过表1记录每个数据中心包含的每个逻辑磁盘单元对应的元数据信息:In practical applications, as shown in Table 1, the metadata information corresponding to each logical disk unit contained in each data center can be recorded through Table 1:
表1 逻辑磁盘单元的元数据信息一览表Table 1 List of metadata information of logical disk units
本申请实施例中,状态信息包括如下信息中的部分或全部:In the embodiment of this application, status information includes some or all of the following information:
1、网络运行状态描述信息;1. Network operating status description information;
2、所述存储区域网络的网络设备运行状态描述信息;2. Description information of the operating status of the network equipment of the storage area network;
3、辅助设备运行状态描述信息。3. Description information of the operating status of auxiliary equipment.
实际应用中,参阅表2所示,可以通过表2记录每个数据中心对应的状态信息:In practical applications, refer to Table 2. The status information corresponding to each data center can be recorded through Table 2:
表2 数据中心的状态信息一览表Table 2 List of status information of data center
步骤320:基于数据描述信息和各个目标信息,确定业务请求对应的目标数据中心,其中,目标数据中心是各个数据中心中的至少一个。Step 320: Determine the target data center corresponding to the service request based on the data description information and each target information, where the target data center is at least one of each data center.
本申请实施例中,在执行步骤320时,通过请求操作类型的不同,执行不同的数据调度流程。In the embodiment of the present application, when step 320 is executed, different data scheduling processes are executed based on different request operation types.
情况一,若请求操作类型是读操作,参阅图5所示,具体流程如下:Case 1: If the requested operation type is a read operation, see Figure 5. The specific process is as follows:
步骤3201:从各个元数据信息中,查询数据描述信息对应的存量数据。Step 3201: Query the existing data corresponding to the data description information from each metadata information.
步骤3202:将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为目标数据中心。Step 3202: Determine the data center corresponding to the queried data center index information of the existing data as the target data center.
情况二,若请求操作类型是写操作,参阅图6所示,具体流程如下:Case 2: If the requested operation type is a write operation, see Figure 6. The specific process is as follows:
步骤3201’:基于数据描述信息和各个状态信息,确定待传输数据对应的调度计划,其中,状态信息是通过任一数据中心对应的状态信息接口定期采集得到的。Step 3201': Determine the scheduling plan corresponding to the data to be transmitted based on the data description information and each status information. The status information is regularly collected through the status information interface corresponding to any data center.
步骤3202’:基于调度计划,确定目标数据中心。Step 3202': Based on the scheduling plan, determine the target data center.
步骤330:将目标数据中心的数据面接口信息返回至客户端,以使客户端基于数据面接口信息构建业务请求对应的上下行数据传输通道。Step 330: Return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
本申请实施例中,在执行步骤320,确定业务请求对应的目标数据中心后,由于跨区域数据调度系统通过调度模块已收集了各个数据中心的数据面接口信息,那么,在执行步骤330时,将目标数据中心的数据面接口信息返回至客户端,这样,客户端既可以根据跨区域数据调度系统返回的数据面接口信息,构建与目标数据中心的上下行数据传输通道,从而实现待传输数据的数据传输。In the embodiment of this application, after executing step 320 and determining the target data center corresponding to the service request, since the cross-regional data scheduling system has collected the data plane interface information of each data center through the scheduling module, then when executing step 330, Return the data plane interface information of the target data center to the client. In this way, the client can build uplink and downlink data transmission channels with the target data center based on the data plane interface information returned by the cross-regional data scheduling system, thereby realizing the data to be transmitted. data transmission.
在实际应用中,通常业务请求同时包含读操作和写操作,那么,参阅图7所示,图7示出了本申请实施例中跨区域数据调度系统通过调度模块进行数据调度的逻辑示意图。In practical applications, usually service requests include both read operations and write operations. Then, refer to Figure 7, which shows a logical schematic diagram of the cross-region data scheduling system in the embodiment of the present application performing data scheduling through the scheduling module.
本申请实施中,参阅图7所示,在执行步骤300之后,执行步骤310之前,需对业务请求包含的请求操作类型进行分类,从而将读操作和写操作区分开,然后,并行执行后续数据调度流程。In the implementation of this application, as shown in Figure 7, after executing step 300 and before executing step 310, the request operation types contained in the business request need to be classified to distinguish read operations and write operations, and then execute subsequent data in parallel. Scheduling process.
具体实施中,若请求操作类型是读操作,则通过调度模块,从元数据服务器中获取各个元数据信息,执行搜索操作,即从各个元数据信息中,查询数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为目标数据中心;若请求操作类型是写操作,则通过调度模块,从元数据服务器中获取状态信息,执行决策操作,即基于数据描述信息和各个状态信息,确定待传输数据对应的调度计划;并基于调度计划,确定目标数据中心。In the specific implementation, if the requested operation type is a read operation, each metadata information is obtained from the metadata server through the scheduling module, and a search operation is performed, that is, the existing data corresponding to the data description information is queried from each metadata information, and The data center corresponding to the data center index information of the queried stock data is determined as the target data center; if the requested operation type is a write operation, the status information is obtained from the metadata server through the scheduling module and the decision-making operation is performed, that is Based on the data description information and each status information, the scheduling plan corresponding to the data to be transmitted is determined; and based on the scheduling plan, the target data center is determined.
然后,将目标数据中心的数据面接口信息返回至客户端,以使客户端基于数据面接口信息构建业务请求对应的上下行数据传输通道,以便于后续顺利传输待传输数据。Then, the data plane interface information of the target data center is returned to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information, so as to facilitate subsequent smooth transmission of the data to be transmitted.
相应的,本申请实施例中,在执行步骤330之后,若业务请求包含的请求操作类型为写操作,则参阅图8所示,进一步执行如下步骤:Accordingly, in this embodiment of the present application, after step 330 is executed, if the request operation type included in the service request is a write operation, refer to Figure 8 to further perform the following steps:
步骤800:在确定写操作完成后,收集目标数据中心的目标元数据信息。Step 800: After determining that the write operation is completed, collect target metadata information of the target data center.
本申请实施例中,在执行步骤800时,在确定写操作完成后,通过调度模块收集目标数据中心的目标元数据信息。In the embodiment of the present application, when step 800 is executed, after it is determined that the write operation is completed, the target metadata information of the target data center is collected through the scheduling module.
步骤810:基于目标元数据信息,更新目标信息中的已有元数据信息。Step 810: Based on the target metadata information, update the existing metadata information in the target information.
本申请实施例中,在执行步骤810时,基于目标元数据信息,更新元数据服务器中已记录(即,已存储)的目标信息中的已有元数据信息,这样,可以保证元数据服务器中存储的元数据信息与各个数据中心的每个逻辑磁盘单元对应的元数据信息一致,便于后续准确、快速执行新接收到的业务请求,从而保证业务的连续性。In the embodiment of the present application, when step 810 is executed, based on the target metadata information, the existing metadata information in the target information that has been recorded (that is, stored) in the metadata server is updated. In this way, it can be ensured that the The stored metadata information is consistent with the metadata information corresponding to each logical disk unit in each data center, which facilitates the subsequent accurate and rapid execution of newly received business requests, thereby ensuring business continuity.
在一些实施例中,由于本申请中,在纳管的每个数据中心中部署了状态信息接口,这样,跨区域数据调度系统可以通过调度模块对某个或某些数据中心中的存量数据进行数据迁移,具体实施中,参阅图9所示,以在第一数据中心和第二数据中心之间执行数据迁移为例,具体数据调度流程包括:In some embodiments, since in this application, a status information interface is deployed in each managed data center, in this way, the cross-regional data scheduling system can use the scheduling module to perform operations on the existing data in one or some data centers. In the specific implementation of data migration, refer to Figure 9, taking data migration between the first data center and the second data center as an example. The specific data scheduling process includes:
步骤900:将第一数据中心的目标存量数据迁移到第二数据中心中,其中,第一数据中心是各个数据中心中的任意一个,第二数据中心是各个数据中心中,除第一数据中心之外的任意一个数据中心。Step 900: Migrate the target stock data of the first data center to the second data center, where the first data center is any one of the data centers, and the second data center is any one of the data centers, except the first data center. any data center outside.
步骤910:在完成数据迁移后,基于第一数据中心的第一元数据信息,以及,第二数据中心的第二元数据信息,更新已有的元数据信息。Step 910: After completing the data migration, update the existing metadata information based on the first metadata information of the first data center and the second metadata information of the second data center.
需要说明的是,上述步骤900-步骤910仅以在第一数据中心和第二数据中心之间执行数据迁移为例,实际应用中,可以在第一数据中心、第二数据中心、第三数据中心之间,甚至更多的数据中心之间进行数据迁移,本申请不做具体限定。It should be noted that the above steps 900 to 910 only take data migration between the first data center and the second data center as an example. In actual applications, the data migration may be performed between the first data center, the second data center, and the third data center. Data migration between centers or even between more data centers is not specifically limited in this application.
例如,将第一数据中心的目标存量数据中的部分目标存量数据,迁移至第二数据中心中,将剩余的目标存量数据迁移到第三数据中心中等等。For example, part of the target inventory data in the first data center is migrated to the second data center, the remaining target inventory data is migrated to the third data center, and so on.
本申请实施例通过的一种跨区域的数据调度方法,通过上述跨区域数据调度系统提升了已搭建存储区域网络架构的存储服务的性能;通过调度模块将若干个数据中心纳管起来,方便客户端与多个数据中心构建业务请求的上下行数据传输通道,从而实现了根据状态信息进行的负载均衡调度;通过上述跨区域数据调度系统也增强了存储服务整体的拓展性,可以通过增减纳管的数据中心的数量,实现并行扩展与缩小。The cross-region data scheduling method adopted in the embodiment of this application improves the performance of storage services of the established storage area network architecture through the above-mentioned cross-region data scheduling system; several data centers are managed through the scheduling module to facilitate customers It builds uplink and downlink data transmission channels for business requests with multiple data centers, thereby realizing load balancing scheduling based on status information; the above cross-regional data scheduling system also enhances the overall scalability of storage services, and can increase or decrease storage capacity by increasing or decreasing capacity. The number of managed data centers enables parallel expansion and contraction.
进一步地,通过调度模块对增量数据和存量数据进行的数据调度和数据迁移,实现了无感知的扩容与迁移,同时,根据元数据信息进行纳管的数据中心的内部存储资源的回收,提高了每个数据中心的基础设施利用率。Furthermore, the data scheduling and data migration of incremental data and stock data through the scheduling module realizes imperceptible expansion and migration. At the same time, the internal storage resources of the managed data center are recycled based on metadata information, improving The infrastructure utilization of each data center.
基于同一发明构思,参阅图10所示,本申请实施例中提供一种跨区域的数据调度装置,包括:Based on the same inventive concept, as shown in Figure 10, an embodiment of the present application provides a cross-region data scheduling device, including:
接收模块1010,用于接收客户端发送的业务请求,其中,所述业务请求包括请求操作类型、待传输数据的数据描述信息;The receiving module 1010 is used to receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
获取模块1020,用于基于所述请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,所述目标信息包括元数据信息和状态信息中的至少一种,所述元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,所述状态信息表征任一数据中心的运行状态描述信息;The acquisition module 1020 is used to obtain the target information of each data center based on the requested operation type. The storage architecture of each data center is built based on the storage area network. The target information includes metadata information and status information. At least one of, the metadata information represents the inventory data description information of each logical disk unit included in any data center, and the status information represents the operating status description information of any data center;
确定模块1030,用于基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,其中,所述目标数据中心是所述各个数据中心中的至少一个;Determining module 1030, configured to determine the target data center corresponding to the service request based on the data description information and each target information, wherein the target data center is at least one of the various data centers;
发送模块1040,用于将所述目标数据中心的数据面接口信息返回至所述客户端,以使所述客户端基于所述数据面接口信息构建所述业务请求对应的上下行数据传输通道。The sending module 1040 is configured to return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
在一些可能的实施例中,若所述请求操作类型是读操作,则所述目标信息是所述元数据信息;若所述请求操作类型是写操作,则所述目标信息是所述状态信息;In some possible embodiments, if the requested operation type is a read operation, the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information. ;
所述基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,所述确定模块1030用于:Determining the target data center corresponding to the service request based on the data description information and each target information, the determination module 1030 is used to:
若所述请求操作类型是读操作,则从所述各个元数据信息中,查询所述数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为所述目标数据中心;If the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
若所述请求操作类型是写操作,则基于所述数据描述信息和所述各个状态信息,确定所述待传输数据对应的调度计划,并基于所述调度计划,确定所述目标数据中心,其中,所述状态信息是通过所述任一数据中心对应的状态信息接口定期采集得到的。If the requested operation type is a write operation, determine the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
在一些可能的实施例中,在所述将所述目标数据中心的数据面接口信息返回至所述客户端之后,所述发送模块1040还用于:In some possible embodiments, after returning the data plane interface information of the target data center to the client, the sending module 1040 is also used to:
在确定所述写操作完成后,收集所述目标数据中心的目标元数据信息;After determining that the write operation is completed, collect target metadata information of the target data center;
基于所述目标元数据信息,更新所述目标信息中的已有元数据信息。Based on the target metadata information, existing metadata information in the target information is updated.
在一些可能的实施例中,所述确定模块1030用于将第一数据中心的目标存量数据迁移到第二数据中心中,其中,所述第一数据中心是所述各个数据中心中的任意一个,所述第二数据中心是所述各个数据中心中,除所述第一数据中心之外的任意一个数据中心;还用于在完成数据迁移后,基于所述第一数据中心的第一元数据信息,以及,所述第二数据中心的第二元数据信息,更新已有的元数据信息。In some possible embodiments, the determining module 1030 is used to migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the respective data centers. , the second data center is any one of the data centers except the first data center; it is also used to, after the data migration is completed, the first element based on the first data center The data information, and the second metadata information of the second data center, update the existing metadata information.
在一些可能的实施例中,所述状态信息包括如下信息中的部分或全部:In some possible embodiments, the status information includes some or all of the following information:
网络运行状态描述信息;Network operating status description information;
所述存储区域网络的网络设备运行状态描述信息;The network device operating status description information of the storage area network;
辅助设备运行状态描述信息。Auxiliary equipment operating status description information.
在一些可能的实施例中,所述元数据信息包括如下信息中的部分或全部:In some possible embodiments, the metadata information includes some or all of the following information:
数据中心索引信息;Data center index information;
每个逻辑磁盘单元的存量数据的数据偏移;Data offset of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的数据长度;The data length of the inventory data of each logical disk unit;
每个逻辑磁盘单元的存量数据的时间戳。The timestamp of the inventory data for each logical disk unit.
参阅图11所示,本申请实施例中提供一种电子设备,包括:处理器1101和存储器1102;Referring to Figure 11, an electronic device is provided in an embodiment of the present application, including: a processor 1101 and a memory 1102;
存储器1102,用于存储处理器1101执行的计算机程序。存储器1102可以是易失性存储器(volatile memory),例如随机存取存储器(random-access memory,RAM);存储器1102也可以是非易失性存储器(non-volatile memory),例如只读存储器,快闪存储器(flash memory),硬盘(hard disk drive,HDD)或固态硬盘(solid-state drive,SSD)、或者存储器1102是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。存储器1102可以是上述存储器的段合。Memory 1102 is used to store computer programs executed by the processor 1101. The memory 1102 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory (non-volatile memory). memory), such as read-only memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD), or the memory 1102 is capable of carrying or storing instructions or data. Without limitation, any other medium that may be in the form of the desired program code and capable of being accessed by a computer. The memory 1102 may be a combination of the above-described memories.
处理器1101,可以包括一个或多个中央处理单元(central processing unit, CPU),图形处理单元(Graphics Processing Unit,GPU)或者为数字处理单元等等。Processor 1101 may include one or more central processing units (central processing units). processing unit (CPU), graphics processing unit (GPU) or digital processing unit, etc.
本申请实施例中不限定上述存储器1102和处理器1101之间的具体连接介质。本申请实施例在图11中以存储器1102和处理器1101之间通过总线1103连接,总线1103在图11中以粗线表示,总线1103可以分为地址总线、数据总线、控制总线等。为便于表示,图11中仅用一条粗线表示,但并不表示仅有一根总线或一种类型的总线。The specific connection medium between the above-mentioned memory 1102 and the processor 1101 is not limited in the embodiment of the present application. In the embodiment of the present application, the memory 1102 and the processor 1101 are connected through a bus 1103 in Figure 11. The bus 1103 is represented by a thick line in Figure 11. The bus 1103 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.
其中,所述存储器存储有程序代码,当所述程序代码被所述处理器1101执行时,使得所述处理器1101执行如上述各个实施例中执行的任意一种方法。The memory stores program code. When the program code is executed by the processor 1101, it causes the processor 1101 to perform any of the methods performed in the above embodiments.
由于该电子设备即是执行本申请实施例中的方法的电子设备,并且该电子设备解决问题的原理与该方法相似,因此该电子设备的实施可以参见方法的实施,重复之处不再赘述。Since the electronic device is an electronic device that performs the method in the embodiment of the present application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation of the method, and repeated details will not be repeated.
基于同一发明构思,本申请实施例提供一种计算机可读存储介质,其上存储有计算机程序指令,该计算机程序指令被处理器执行时实现上述各个实施例中的任意一种方法。Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, any one of the methods in the above embodiments can be implemented.
本领域内的技术人员应明白,本申请的实施例可提供为方法、系统、或计算机程序产品。因此,本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且,本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本申请是参照根据本申请的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器,使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图中的一个流程或多个流程和/或方框图中的一个方框或多个方框中指定的功能的装置。The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use Means for implementing the functions specified in one process or processes in the flowchart and/or one block or multiple blocks in the block diagram.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中,使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品,该指令装置实现在流程图中的一个流程或多个流程和/或方框图中的一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in the process or processes in the flowchart and/or the block or blocks in the block diagram.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上,使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理,从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图中的一个流程或多个流程和/或方框图中的一个方框或多个方框中指定的功能的步骤。These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes in a flowchart diagram and/or in a block or blocks in a block diagram.
显然,本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样,倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内,则本申请也意图包含这些改动和变型在内。Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and equivalent technologies, the present application is also intended to include these modifications and variations.

Claims (10)

  1. 一种跨区域的数据调度方法,其特征在于,包括:A cross-region data scheduling method is characterized by including:
    接收客户端发送的业务请求,其中,所述业务请求包括请求操作类型、待传输数据的数据描述信息;Receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
    基于所述请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,所述目标信息包括元数据信息和状态信息中的至少一种,所述元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,所述状态信息表征任一数据中心的运行状态描述信息;Based on the request operation type, the target information of each data center is obtained, where the storage architecture of each data center is built based on the storage area network, and the target information includes at least one of metadata information and status information, so The metadata information represents the inventory data description information of each logical disk unit contained in any data center, and the status information represents the operating status description information of any data center;
    基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,其中,所述目标数据中心是所述各个数据中心中的至少一个;Based on the data description information and each target information, determine the target data center corresponding to the service request, wherein the target data center is at least one of the various data centers;
    将所述目标数据中心的数据面接口信息返回至所述客户端,以使所述客户端基于所述数据面接口信息构建所述业务请求对应的上下行数据传输通道。The data plane interface information of the target data center is returned to the client, so that the client constructs an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  2. 如权利要求1所述的方法,其特征在于,若所述请求操作类型是读操作,则所述目标信息是所述元数据信息;若所述请求操作类型是写操作,则所述目标信息是所述状态信息;The method of claim 1, wherein if the requested operation type is a read operation, the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information;
    所述基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,包括:Determining the target data center corresponding to the service request based on the data description information and each target information includes:
    若所述请求操作类型是读操作,则从所述各个元数据信息中,查询所述数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为所述目标数据中心;If the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
    若所述请求操作类型是写操作,则基于所述数据描述信息和所述各个状态信息,确定所述待传输数据对应的调度计划,并基于所述调度计划,确定所述目标数据中心,其中,所述状态信息是通过所述任一数据中心对应的状态信息接口定期采集得到的。If the requested operation type is a write operation, determine the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
  3. 如权利要求2所述的方法,其特征在于,在所述将所述目标数据中心的数据面接口信息返回至所述客户端之后,还包括:The method of claim 2, wherein after returning the data plane interface information of the target data center to the client, it further includes:
    在确定所述写操作完成后,收集所述目标数据中心的目标元数据信息;After determining that the write operation is completed, collect target metadata information of the target data center;
    基于所述目标元数据信息,更新所述目标信息中的已有元数据信息。Based on the target metadata information, existing metadata information in the target information is updated.
  4. 如权利要求1所述的方法,其特征在于,所述方法包括:The method of claim 1, wherein the method includes:
    将第一数据中心的目标存量数据迁移到第二数据中心中,其中,所述第一数据中心是所述各个数据中心中的任意一个,所述第二数据中心是所述各个数据中心中,除所述第一数据中心之外的任意一个数据中心;Migrate the target inventory data of the first data center to a second data center, where the first data center is any one of the data centers, and the second data center is one of the data centers, Any data center except the first data center;
    在完成数据迁移后,基于所述第一数据中心的第一元数据信息,以及,所述第二数据中心的第二元数据信息,更新已有的元数据信息。After the data migration is completed, the existing metadata information is updated based on the first metadata information of the first data center and the second metadata information of the second data center.
  5. 如权利要求1-4任一所述的方法,其特征在于,所述状态信息包括如下信息中的部分或全部:The method according to any one of claims 1 to 4, characterized in that the status information includes part or all of the following information:
    网络运行状态描述信息;Network operating status description information;
    所述存储区域网络的网络设备运行状态描述信息;The network device operating status description information of the storage area network;
    辅助设备运行状态描述信息。Auxiliary equipment operating status description information.
  6. 如权利要求1-4任一所述的方法,其特征在于,所述元数据信息包括如下信息中的部分或全部:The method according to any one of claims 1 to 4, characterized in that the metadata information includes part or all of the following information:
    数据中心索引信息;Data center index information;
    每个逻辑磁盘单元的存量数据的数据偏移;Data offset of the inventory data of each logical disk unit;
    每个逻辑磁盘单元的存量数据的数据长度;The data length of the inventory data of each logical disk unit;
    每个逻辑磁盘单元的存量数据的时间戳。The timestamp of the inventory data for each logical disk unit.
  7. 一种跨区域的数据调度装置,其特征在于,包括:A cross-regional data scheduling device, characterized by including:
    接收模块,用于接收客户端发送的业务请求,其中,所述业务请求包括请求操作类型、待传输数据的数据描述信息;A receiving module, configured to receive a service request sent by the client, where the service request includes the requested operation type and data description information of the data to be transmitted;
    获取模块,用于基于所述请求操作类型,获取各个数据中心的目标信息,其中,每个数据中心的存储架构是基于存储区域网络搭建的,所述目标信息包括元数据信息和状态信息中的至少一种,所述元数据信息表征任一数据中心包含的每个逻辑磁盘单元的存量数据描述信息,所述状态信息表征任一数据中心的运行状态描述信息;The acquisition module is used to obtain the target information of each data center based on the request operation type. The storage architecture of each data center is built based on the storage area network. The target information includes metadata information and status information. At least one, the metadata information represents the inventory data description information of each logical disk unit included in any data center, and the status information represents the operating status description information of any data center;
    确定模块,用于基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,其中,所述目标数据中心是所述各个数据中心中的至少一个;Determining module, configured to determine the target data center corresponding to the service request based on the data description information and each target information, wherein the target data center is at least one of the various data centers;
    发送模块,用于将所述目标数据中心的数据面接口信息返回至所述客户端,以使所述客户端基于所述数据面接口信息构建所述业务请求对应的上下行数据传输通道。A sending module, configured to return the data plane interface information of the target data center to the client, so that the client can construct an uplink and downlink data transmission channel corresponding to the service request based on the data plane interface information.
  8. 如权利要求7所述的装置,其特征在于,若所述请求操作类型是读操作,则所述目标信息是所述元数据信息;若所述请求操作类型是写操作,则所述目标信息是所述状态信息;The device of claim 7, wherein if the requested operation type is a read operation, the target information is the metadata information; if the requested operation type is a write operation, the target information is the status information;
    所述基于所述数据描述信息和各个目标信息,确定所述业务请求对应的目标数据中心,所述确定模块用于:Determine the target data center corresponding to the service request based on the data description information and each target information, and the determination module is used to:
    若所述请求操作类型是读操作,则从所述各个元数据信息中,查询所述数据描述信息对应的存量数据,并将查询到的存量数据的数据中心索引信息所对应的数据中心,确定为所述目标数据中心;If the requested operation type is a read operation, query the inventory data corresponding to the data description information from each metadata information, and determine the data center corresponding to the data center index information of the queried inventory data. For the target data center;
    若所述请求操作类型是写操作,则基于所述数据描述信息和所述各个状态信息,确定所述待传输数据对应的调度计划,并基于所述调度计划,确定所述目标数据中心,其中,所述状态信息是通过所述任一数据中心对应的状态信息接口定期采集得到的。If the requested operation type is a write operation, determine the scheduling plan corresponding to the data to be transmitted based on the data description information and the respective status information, and determine the target data center based on the scheduling plan, where , the status information is collected regularly through the status information interface corresponding to any data center.
  9. 一种电子设备,其特征在于,所述电子设备包括处理器和存储器,An electronic device, characterized in that the electronic device includes a processor and a memory,
    所述存储器,用于存储计算机程序或指令;The memory is used to store computer programs or instructions;
    所述处理器,用于执行存储器中的计算机程序或指令,使得权利要求1-6中任一项所述的方法被执行。The processor is configured to execute computer programs or instructions in the memory, so that the method described in any one of claims 1-6 is executed.
  10. 一种计算机可读存储介质,其上存储有计算机程序指令,其特征在于,该计算机程序指令被处理器执行时实现权利要求1-6中任一项所述方法的步骤。A computer-readable storage medium with computer program instructions stored thereon, characterized in that when the computer program instructions are executed by a processor, the steps of the method described in any one of claims 1-6 are implemented.
PCT/CN2022/141408 2022-07-28 2022-12-23 Cross-region data scheduling method and apparatus, device, and storage medium WO2024021470A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210899880.5 2022-07-28
CN202210899880.5A CN115292280A (en) 2022-07-28 2022-07-28 Cross-region data scheduling method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
WO2024021470A1 true WO2024021470A1 (en) 2024-02-01

Family

ID=83823298

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/141408 WO2024021470A1 (en) 2022-07-28 2022-12-23 Cross-region data scheduling method and apparatus, device, and storage medium

Country Status (2)

Country Link
CN (1) CN115292280A (en)
WO (1) WO2024021470A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118018549A (en) * 2024-02-06 2024-05-10 中科云谷科技有限公司 Method, processor, device and system for constructing cross-regional service system

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115292280A (en) * 2022-07-28 2022-11-04 天翼云科技有限公司 Cross-region data scheduling method, device, equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110233865A (en) * 2018-03-06 2019-09-13 阿里巴巴集团控股有限公司 Trans-regional service calling method, device and system
CN111064802A (en) * 2019-12-26 2020-04-24 北京奇艺世纪科技有限公司 Network request processing method and device, electronic equipment and storage medium
JP2021002207A (en) * 2019-06-21 2021-01-07 Necソリューションイノベータ株式会社 Information carrying device, backup device, information carrying method, backup method, program and recording medium
CN113992680A (en) * 2021-11-10 2022-01-28 中国工商银行股份有限公司 Scheduling method, device, equipment and medium applied to distributed multi-activity system
CN115292280A (en) * 2022-07-28 2022-11-04 天翼云科技有限公司 Cross-region data scheduling method, device, equipment and storage medium

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110233865A (en) * 2018-03-06 2019-09-13 阿里巴巴集团控股有限公司 Trans-regional service calling method, device and system
JP2021002207A (en) * 2019-06-21 2021-01-07 Necソリューションイノベータ株式会社 Information carrying device, backup device, information carrying method, backup method, program and recording medium
CN111064802A (en) * 2019-12-26 2020-04-24 北京奇艺世纪科技有限公司 Network request processing method and device, electronic equipment and storage medium
CN113992680A (en) * 2021-11-10 2022-01-28 中国工商银行股份有限公司 Scheduling method, device, equipment and medium applied to distributed multi-activity system
CN115292280A (en) * 2022-07-28 2022-11-04 天翼云科技有限公司 Cross-region data scheduling method, device, equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118018549A (en) * 2024-02-06 2024-05-10 中科云谷科技有限公司 Method, processor, device and system for constructing cross-regional service system

Also Published As

Publication number Publication date
CN115292280A (en) 2022-11-04

Similar Documents

Publication Publication Date Title
US10412170B2 (en) Retention-based data management in a network-based data store
WO2024021470A1 (en) Cross-region data scheduling method and apparatus, device, and storage medium
US20130332612A1 (en) Transmission of map/reduce data in a data center
US20140068224A1 (en) Block-level Access to Parallel Storage
US8135918B1 (en) Data de-duplication for iSCSI
CN107832423B (en) File reading and writing method for distributed file system
US9110820B1 (en) Hybrid data storage system in an HPC exascale environment
US9983823B1 (en) Pre-forking replicas for efficient scaling of a distribued data storage system
CN109933312B (en) Method for effectively reducing I/O consumption of containerized relational database
CN114201421B (en) Data stream processing method, storage control node and readable storage medium
US11625192B2 (en) Peer storage compute sharing using memory buffer
US8386741B2 (en) Method and apparatus for optimizing data allocation
US10922272B2 (en) Providing data across multiple clusters in a single namespace
WO2021213281A1 (en) Data reading method and system
CN110119304A (en) A kind of interruption processing method, device and server
CN111159176A (en) Method and system for storing and reading mass stream data
CN113806300B (en) Data storage method, system, device, equipment and storage medium
CN115129621B (en) Memory management method, device, medium and memory management module
US11347413B2 (en) Opportunistic storage service
US20240036728A1 (en) Method and apparatus for processing data, reduction server, and mapping server
WO2024001863A1 (en) Data processing method and related device
US11301306B2 (en) Apparatus, systems, and methods for remote system batch processing
WO2023050856A1 (en) Data processing method and storage system
US11093493B1 (en) Dynamically switching between query and scan for optimizing table reads
KR20140061194A (en) Network distributed file system and method using iscsi storage system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22952901

Country of ref document: EP

Kind code of ref document: A1