CN111510493A - Distributed data transmission method and device - Google Patents
Distributed data transmission method and device Download PDFInfo
- Publication number
- CN111510493A CN111510493A CN202010294993.3A CN202010294993A CN111510493A CN 111510493 A CN111510493 A CN 111510493A CN 202010294993 A CN202010294993 A CN 202010294993A CN 111510493 A CN111510493 A CN 111510493A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- target
- processing server
- transfer processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a distributed data transmission method and a device, wherein the method comprises the following steps: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the volume value of the target transmission data set and the authentication identification character string of the data source server; judging whether the sum of the residual space of each transfer processing server is larger than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one transfer processing server as the target transfer processing server set so that each transfer processing server in the target transfer processing server set receives each target transmission data sent by the data source server respectively and in parallel, and segmenting or merging the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data; and respectively sending the file packets to a target data receiving server. The method and the device can improve the efficiency and the reliability of data transmission on the basis of realizing parallel data transmission.
Description
Technical Field
The present application relates to the field of distributed architecture technologies, and in particular, to a distributed data transmission method and apparatus.
Background
In a distributed architecture, data is typically split into multiple small units and stored on different servers according to a distribution key. In a disaster recovery scenario, it is necessary to merge and backup dispersed data to a centralized backup server, or to distribute data on the centralized backup server to corresponding distributed servers.
Existing financial systems back up or restore data on a source server through an interface. Under a distributed scene, data cannot be monitored and processed in a transmission process, unified scheduling is lacked, operations such as flow control, multi-server concurrent scheduling control, data conversion, data merging and the like cannot be performed, once a backup process is started, intervention cannot be performed, controllability and flexible diversity of a data backup or data recovery process are affected, and reliability of data transmission is low.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a distributed data transmission method and a distributed data transmission device, which can improve the efficiency and reliability of data transmission on the basis of realizing parallel data transmission.
In order to solve the technical problem, the present application provides the following technical solutions:
in a first aspect, the present application provides a distributed data transmission method, including:
receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the volume value of the target transmission data set and the authentication identification character string of the data source server;
judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and parallelly, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and respectively sending the file packets to a target data receiving server.
Further, before the receiving the data transmission request sent by the data source server, the method further includes: receiving user authentication information sent by the data source server; and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
Further, after the sending the file packets to the target data receiving servers respectively, the method further includes: and storing the target transmission data group to the target data receiving server.
Further, in the process that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the method further includes: and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, if so, adjusting the flow to the flow threshold.
Further, the taking at least one corresponding relay processing server as a target relay processing server group according to the capacity value of the target transmission data group, so that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, includes: and respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
In a second aspect, the present application provides a distributed data transmission apparatus, including:
a receiving request module, configured to receive a data transmission request sent by a data source server, where the data transmission request includes: the volume value of the target transmission data set and the authentication identification character string of the data source server;
the processing module is used for judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and parallelly, and performs segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and the sending module is used for respectively sending the file packets to the target data receiving server.
Further, the distributed data transmission apparatus further includes: the receiving authentication information module is used for receiving the user authentication information sent by the data source server; and the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning the authentication identification character string corresponding to the transfer processing server to the data source server.
Further, the distributed data transmission apparatus further includes: and the storage module is used for storing the target transmission data group to the target data receiving server.
Further, in the process that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the processing module is further configured to determine whether a traffic of the target transmission data sent to the corresponding relay processing server exceeds a traffic threshold, and if so, adjust the traffic to the traffic threshold.
Further, the processing module includes: and the target transmission data acquisition unit is used for respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the distributed data transmission method when executing the program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer instructions that, when executed, implement the distributed data transmission method.
According to the technical scheme, the application provides a distributed data transmission method and device. Wherein, the method comprises the following steps: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the volume value of the target transmission data set and the authentication identification character string of the data source server; judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and parallelly, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data; and respectively sending the file packets to a target data receiving server. The method and the device can improve the efficiency and the reliability of data transmission on the basis of realizing parallel data transmission; specifically, the data transmission process can be monitored in real time, such as flow monitoring and file content blacklist monitoring; the data are flexibly scheduled, parallel data transmission can be realized, the utilization rate of each server is improved, and the cost is saved; by splitting or merging the data, the reliability and controllability of data transmission can be improved, and meanwhile, the accuracy of determining the wrong node can be improved; the transfer processing server is expandable, and different numbers of data interception processors can be expanded according to business needs; the data transmission can be controlled in a customized and refined mode, and the data flow after being transferred can be controlled according to a predefined method, such as flow control and file content blacklist control.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a distributed data transmission method according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a distributed data transmission method according to another embodiment of the present application;
fig. 3 is a schematic structural diagram of a distributed data transmission apparatus according to an embodiment of the present application;
FIG. 4 is a topology diagram of a coordination server and a transit server according to an embodiment of the present application;
FIG. 5 is a data source server and transit server topology diagram of an exemplary application of the present application;
FIG. 6 is a topology diagram of a transit server and a backup storage server according to an embodiment of the present disclosure;
FIG. 7 is a schematic structural diagram of a transit server according to an exemplary embodiment of the present application;
FIG. 8 is a schematic diagram of a data source server according to an exemplary embodiment of the present application;
FIG. 9 is a flowchart of data backup according to an embodiment of the present application;
FIG. 10 is a flow chart of data recovery according to an embodiment of the present application;
fig. 11 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Based on this, in order to improve the efficiency and reliability of data transmission on the basis of realizing parallel data transmission, an embodiment of the present application provides a distributed data transmission apparatus, which may be a server or a client device, where the client device may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, an intelligent wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch and intelligent bracelet etc..
In practical applications, part of the distributed data transmission may be performed on the server side as described above, or all operations may be performed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
The server and the client device may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (Remote Procedure Call Protocol), a REST Protocol (Representational State Transfer Protocol), and the like used above the above Protocol.
The following examples are intended to illustrate the details.
In order to improve the efficiency and reliability of data transmission based on the realization of parallel data transmission, the embodiment provides a distributed data transmission method in which the execution subject is a distributed data transmission apparatus, wherein the distributed data transmission apparatus may be a single server; or a transmission device combining the coordination server and the transfer processing server; or at least one of the transit processing servers, where the transit processing server stores basic information of each transit processing server, such as remaining space information and addresses; as shown in fig. 1, the method specifically includes the following steps:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the capacity value of the target transmission data set and the authentication identification character string of the data source server.
Specifically, the authentication identification character string is a unique code of the data source server, and the data source server sending the data transmission request and the transfer processing server matched with the data source server can be determined according to the authentication identification character string. The target transmission data may be a financial transaction data file or a customer information file, etc.
S200: and judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and in parallel, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data.
Specifically, the capacity value of each file packet does not exceed the threshold value. The remaining space is the allocable resource of the transit processing server. The number of the transfer processing servers in the target transfer processing server group is the same as the number of the target transmission data. The threshold value can be set according to actual conditions.
It can be understood that, if the sum of the remaining spaces of the corresponding transfer processing servers is judged to be smaller than the capacity value of the target transmission data according to the authentication identification character string, a space shortage prompt message is sent to the data source server.
S300: and respectively sending the file packets to a target data receiving server.
Specifically, the data transmission method provided by the embodiment of the present application is based on a distributed framework for data transmission.
In an embodiment, the determining whether the sum of the remaining spaces of the corresponding transfer processing servers is greater than the capacity value of the target transmission data set according to the authentication identifier string includes:
and determining a transfer processing server matched with the data source server according to the authentication identification character string.
And judging whether the sum of the residual spaces of the transfer processing server is larger than the capacity value of the target transmission data set.
In an embodiment, the taking the corresponding at least one relay processing server as the target relay processing server group according to the capacity value of the target transmission data group includes:
and sequencing the transit processing servers according to a preset sequencing rule, and taking the transit processing servers with the least number and the sum of the residual spaces obtained from front to back according to the sequencing result as the target transit processing server group.
Wherein the size of the target transmission data is related to the remaining space of each of the relay processing servers. The preset ordering rule can be set according to actual needs, and the application is not limited to this. For example, the sorting rule is that the relay processing servers are sorted from large to small according to the respective remaining space or the relay processing servers are sorted from front to back according to the respective starting time. In one example, the transit processing servers with the remaining space are sorted from front to back according to the respective starting time, and the minimum number of transit processing servers with the sum of the remaining space larger than the capacity value of the target transmission data is obtained from front to back.
In an embodiment, before the respectively performing the splitting or the merging on the target transmission data according to the same threshold, the method further includes: and labeling each target transmission data according to the authentication identification character string. And determining a corresponding data source server according to the labeled target transmission data.
In order to further improve the reliability and security of data transmission, as shown in fig. 2, in an embodiment of the present application, before step S100, the method further includes:
s101: and receiving the user authentication information sent by the data source server.
S102: and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
Specifically, the distributed data transmission device performs identity verification on each to-be-matched relay processing server according to the user authentication information, and if the verification passes, the corresponding to-be-matched relay processing server is a relay processing server matched with the data source server.
In order to implement parallel data backup, in an embodiment of the present application, after step S300, the method further includes:
and storing the target transmission data group to the target data receiving server.
In order to implement real-time monitoring on data transmission and improve reliability of data transmission, in an embodiment of the present application, in a process that each relay processing server in the target relay processing server group receives each target transmission data in a target transmission data group sent by the data source server respectively and in parallel, the method further includes:
and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, if so, adjusting the flow to the flow threshold.
The flow threshold may be set according to actual needs, which is not limited in this application.
Specifically, the distributed data transmission apparatus may further monitor whether the target transmission data is in a preset file content blacklist in real time, and if so, stop the transmission process. In addition, the distributed data transmission device can also monitor the process of splitting or merging the target transmission data in real time, and restart the processing process if the processing fails.
In order to further improve the accuracy and efficiency of data transmission, in an embodiment of the present application, the step S200 of using at least one corresponding relay processing server as a target relay processing server group according to the capacity value of the target transmission data group, so that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, includes:
s211: and respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
From the software aspect, in order to improve the efficiency and reliability of data transmission based on the implementation of parallel data transmission, the present application provides an embodiment of a distributed data transmission apparatus for implementing all or part of the contents in the distributed data transmission method, and referring to fig. 3, the distributed data transmission apparatus specifically includes the following contents:
a receiving request module 10, configured to receive a data transmission request sent by a data source server, where the data transmission request includes: the capacity value of the target transmission data set and the authentication identification character string of the data source server.
And the processing module 20 is configured to determine whether the sum of the remaining space of each corresponding relay processing server is greater than the capacity value of the target transmission data set according to the authentication identifier string, if so, use at least one corresponding relay processing server as the target relay processing server set according to the capacity value of the target transmission data set, so that each relay processing server in the target relay processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and in parallel, and perform segmentation or merging processing on the target transmission data according to the same threshold value, so as to obtain each file packet corresponding to the target transmission data.
And the sending module 30 is configured to send the file packages to the target data receiving servers respectively.
In an embodiment of the present application, the distributed data transmission apparatus further includes:
and the receiving authentication information module is used for receiving the user authentication information sent by the data source server.
And the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning the authentication identification character string corresponding to the transfer processing server to the data source server.
In an embodiment of the present application, the distributed data transmission apparatus further includes: and the storage module is used for storing the target transmission data group to the target data receiving server.
In an embodiment of the application, in a process that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the processing module is further configured to determine whether a traffic of the target transmission data sent to the corresponding relay processing server exceeds a traffic threshold, and if so, adjust the traffic to the traffic threshold.
In an embodiment of the present application, the processing module includes:
and the target transmission data acquisition unit is used for respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
The embodiments of the distributed data transmission apparatus provided in this specification may be specifically used to execute the processing flow of the embodiments of the distributed data transmission method, and the functions of the processing flow are not described herein again, and reference may be made to the detailed description of the embodiments of the distributed data transmission method.
In order to improve the data scheduling, controlling, monitoring and processing capabilities in the distributed data transmission and further improve the data transmission efficiency and reliability, the application further provides a specific application example of the distributed data transmission system, and in the specific application example, the specific distributed data transmission system is composed of a coordination server 1, a transfer server 2, a data source server 3 and a backup storage data device 4. The relay server 2 is configured to implement the functions of the relay processing server and/or the distributed data transmission apparatus, and the backup storage server 4 is configured to implement the functions of the target data receiving server, which are described in detail as follows:
(1) as shown in fig. 4, the coordination server 1 is a control device for coordinating and controlling the whole backup transfer process, and includes two servers, and has high availability; the coordination server comprises a high availability module 101 and a communication module 102.
A high availability module 101, configured to, when the primary server is available, the primary server is an active server and the standby server is inactive; when the main server is down, the standby server is upgraded to be the active server, and the management rights of all the transit servers are handed over to the existing active server.
And the communication module 102 is configured to coordinate network communication between the server and the relay server 2.
(2) As shown in fig. 5 to fig. 7, the relay server 2 is a carrying device for data transmission processing, and includes a data pulling module 201, a data pushing module 202, a data backup server module 301, and a relay function module 302.
The data pulling module 201 is configured to pull data to be backed up from a plurality of data source servers according to a real-time load.
And the data pushing module 202 is configured to push the processed backup data to the backup storage server for storage according to the data processing progress, or restore the backup data to the transit server for processing.
The data backup server module 301 is configured to perform a data processing process.
The relay function 302 includes a data receiver, a data monitor, a data processor, and a data transmitter. The data receiver is used for receiving data sent by the data source server or the backup storage server, the data monitor is used for monitoring and managing data streams, the data processor is used for converting the data, and the data sender is used for sending the data to the backup storage data device or the data source server.
(3) Referring to fig. 8, the data source server 3 is a server storing data to be backed up, and includes a data backup client module 401 and a data sending and receiving module 402.
And the data backup client module 401 is configured to send and receive data.
The data sending and receiving module 402 includes a data sender and a data receiver, which are respectively used for sending data to the data backup server on the transit server and receiving data from the data backup server.
(4) The backup storage server 4 is used for storing data.
As can be seen from the above description, in this specific application example, the interceptor is extensible; the data transmission can be finely controlled, and the transferred data flow can be controlled according to a predefined method, such as flow control and file content blacklist control; the scale of the transfer cluster can be transversely expanded and reduced, and the transfer cluster can be expanded and reduced according to the scales of the data source server and the backup storage server, so that resources are reasonably used.
To further explain the present solution, in combination with the distributed data transmission system in the above specific application example, the present application also provides a specific application example of the distributed data backup method, as shown in fig. 9, which specifically includes the following contents:
step 501: and the backup client performs identity authentication, the identity authentication returns an authentication identification character string through the backup server, the effective time is set, and a subsequent request of the backup client needs to carry the authentication identification character string as a request parameter.
Specifically, the backup client initiates an identity authentication request to the backup server. The validity time is a validity time set in the transit server. The authentication identification character string is used for the authentication identification of the backup server side to the backup client side.
Step 502: the backup client applies for resources from the backup server, the backup server distributes the resources according to the current resources, and if the application resources of the backup client are larger than the highest distributable resources, the resources are not enough; otherwise, returning the resource details distributed to the backup client. The backup client may send the file to a designated directory returned by the server.
Specifically, the resource detail includes the available transit server, the available directory and the valid time that have been applied for allocation. And the effective time represents the maximum value of time consumed by the backup server side for sending the resource details to the receiving of the file sent by the backup client side, and if the consumed time exceeds the effective time, the corresponding data transmission task is ended.
Step 503: and the backup client sends the file to the applied transfer server.
Step 504: and intercepting the file by the backup server, processing the file according to the function on the processing chain and monitoring in real time. The backup server intercepts files sent by the backup client, and comprises file format conversion and file segmentation and merging, wherein the file format conversion is to print invisible labels on the files for file identification, the file segmentation operation is to merge small files into 300G files, and to segment large files into a plurality of 300G files.
Step 505: and the backup server sends the processed file to a backup storage server for storage.
Step 506: and the backup server side clears the temporary file generated by the backup processing.
The temporary files are intermediate files generated in the intercepting process and files which are transmitted by the backup client and successfully stored in the backup storage server and received by the transfer server. Such as a log file.
To further explain the present solution, in combination with the distributed data transmission system in the above specific application example, the present application also provides a specific application example of the distributed data recovery method, as shown in fig. 10, which specifically includes the following contents:
step 601: and the backup client performs identity authentication, the identity authentication returns an authentication identification character string through the backup server, the effective time is set, and a subsequent request of a caller needs to carry the authentication identification character string as a request parameter.
Step 602: the backup client applies for resources from the backup server, the backup server distributes the resources according to the current resources, and if the application resources of the backup client are larger than the highest distributable resources, the resources are not enough; otherwise, returning the resource details distributed to the backup client. The backup client may send the file to a designated directory returned by the server.
Step 603: the backup server side requests the files from the backup storage server and receives the files to the designated directory.
Step 604: and intercepting the file by the backup server, processing the file according to the function on the processing chain and monitoring.
Step 605: and the backup server sends the processed file to the backup client for recovery.
Step 606: and the backup server side cleans the temporary file generated by the recovery processing.
As can be seen from the above description, the distributed data transmission method and apparatus provided by the present application can improve the efficiency and reliability of data transmission on the basis of implementing parallel data transmission; specifically, the method and the device can improve the data state monitoring, data conversion and data scheduling capabilities in the data transmission process, improve the data processing efficiency by utilizing the distributed architecture, realize the backup of scattered data concentrated data to the intensive backup storage server, and ensure the consistency of the data.
In terms of hardware, in order to improve efficiency and reliability of data transmission based on parallel data transmission, the present application provides an embodiment of an electronic device for implementing all or part of contents in the distributed data transmission method, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission among the distributed data transmission device, the user terminal and other related equipment; the electronic device may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the electronic device may be implemented with reference to the embodiment for implementing the distributed data transmission method and the embodiment for implementing the distributed data transmission apparatus in the embodiments, and the contents thereof are incorporated herein, and repeated details are not repeated here.
Fig. 11 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 11, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 11 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one or more embodiments of the present application, the distributed data transfer functionality can be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the capacity value of the target transmission data set and the authentication identification character string of the data source server.
S200: and judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and in parallel, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data.
S300: and respectively sending the file packets to a target data receiving server.
As can be seen from the foregoing description, the electronic device provided in the embodiments of the present application can improve the efficiency and reliability of data transmission based on parallel data transmission.
In another embodiment, the distributed data transmission apparatus may be configured separately from the central processor 9100, for example, the distributed data transmission apparatus may be configured as a chip connected to the central processor 9100, and the distributed data transmission function is realized by the control of the central processor.
As shown in fig. 11, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 11; in addition, the electronic device 9600 may further include components not shown in fig. 11, which may be referred to in the prior art.
As shown in fig. 11, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
An input unit 9120 provides input to the cpu 9100, the input unit 9120 is, for example, a key or a touch input device, a power supply 9170 supplies power to the electronic apparatus 9600, a display 9160 displays display objects such as images and characters, and the display may be, for example, an L CD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
As can be seen from the above description, the electronic device provided in the embodiments of the present application can improve the efficiency and reliability of data transmission on the basis of implementing parallel data transmission.
Embodiments of the present application further provide a computer-readable storage medium capable of implementing all steps in the distributed data transmission method in the foregoing embodiments, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all steps of the distributed data transmission method in the foregoing embodiments, for example, when the processor executes the computer program, the processor implements the following steps:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the capacity value of the target transmission data set and the authentication identification character string of the data source server.
S200: and judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and in parallel, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data.
S300: and respectively sending the file packets to a target data receiving server.
As can be seen from the foregoing description, the computer-readable storage medium provided in the embodiments of the present application can improve the efficiency and reliability of data transmission on the basis of implementing parallel data transmission.
In the present application, each embodiment of the method is described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Reference is made to the description of the method embodiments.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the present application are explained by applying specific embodiments in the present application, and the description of the above embodiments is only used to help understanding the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (12)
1. A distributed data transmission method, comprising:
receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the volume value of the target transmission data set and the authentication identification character string of the data source server;
judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and parallelly, and performing segmentation or merging processing on the target transmission data respectively according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and respectively sending the file packets to a target data receiving server.
2. The distributed data transmission method according to claim 1, further comprising, before the receiving the data transmission request sent by the data source server:
receiving user authentication information sent by the data source server;
and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
3. The distributed data transmission method according to claim 1, further comprising, after the sending the file packets to the target data receiving servers, respectively:
and storing the target transmission data group to the target data receiving server.
4. The distributed data transmission method according to claim 1, wherein in the process that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the method further comprises:
and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, if so, adjusting the flow to the flow threshold.
5. The distributed data transmission method according to claim 1, wherein the step of using the corresponding at least one relay processing server as a target relay processing server group according to the capacity value of the target transmission data group, so that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, comprises:
and respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
6. A distributed data transmission apparatus, comprising:
a receiving request module, configured to receive a data transmission request sent by a data source server, where the data transmission request includes: the volume value of the target transmission data set and the authentication identification character string of the data source server;
the processing module is used for judging whether the sum of the residual space of each corresponding transfer processing server is greater than the capacity value of the target transmission data set or not according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives each target transmission data in the target transmission data set sent by the data source server respectively and parallelly, and performs segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and the sending module is used for respectively sending the file packets to the target data receiving server.
7. The distributed data transmission apparatus of claim 6, further comprising:
the receiving authentication information module is used for receiving the user authentication information sent by the data source server;
and the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning the authentication identification character string corresponding to the transfer processing server to the data source server.
8. The distributed data transmission apparatus of claim 6, further comprising:
and the storage module is used for storing the target transmission data group to the target data receiving server.
9. The distributed data transmission apparatus according to claim 6, wherein in a process that each relay processing server in the target relay processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the processing module is further configured to determine whether a traffic sent by the target transmission data to the corresponding relay processing server exceeds a traffic threshold, and if so, adjust the traffic to the traffic threshold.
10. The distributed data transmission apparatus of claim 6, wherein the processing module comprises:
and the target transmission data acquisition unit is used for respectively sending the address of each transfer processing server in the target transfer processing server group, the corresponding storage directory and the residual space information to the data source server so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address of each transfer processing server and the corresponding storage directory.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the distributed data transmission method of any one of claims 1 to 5 when executing the program.
12. A computer readable storage medium having computer instructions stored thereon which, when executed, implement the distributed data transmission method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010294993.3A CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010294993.3A CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111510493A true CN111510493A (en) | 2020-08-07 |
| CN111510493B CN111510493B (en) | 2023-09-26 |
Family
ID=71876136
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010294993.3A Active CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111510493B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860780A (en) * | 2021-03-30 | 2021-05-28 | 中国工商银行股份有限公司 | Data export method and device and terminal equipment |
| CN114679486A (en) * | 2022-03-29 | 2022-06-28 | 网易（杭州）网络有限公司 | Communication interaction method and system, application terminal, client terminal and transfer server terminal |
| CN113283803B (en) * | 2021-06-17 | 2024-04-23 | 金蝶软件（中国）有限公司 | Method for making material demand plan, related device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911772A (en) * | 2017-02-20 | 2017-06-30 | 联想(北京)有限公司 | Server-assignment method, server-assignment device and electronic equipment |
| WO2019075978A1 (en) * | 2017-10-16 | 2019-04-25 | 平安科技（深圳）有限公司 | Data transmission method and apparatus, computer device, and storage medium |
| CN110443695A (en) * | 2019-07-31 | 2019-11-12 | 中国工商银行股份有限公司 | Data processing method and its device, electronic equipment and medium |
| CN110719245A (en) * | 2018-07-11 | 2020-01-21 | 中国移动通信集团浙江有限公司 | Internet of things data transmission method and system |
-
2020
- 2020-04-15 CN CN202010294993.3A patent/CN111510493B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911772A (en) * | 2017-02-20 | 2017-06-30 | 联想(北京)有限公司 | Server-assignment method, server-assignment device and electronic equipment |
| WO2019075978A1 (en) * | 2017-10-16 | 2019-04-25 | 平安科技（深圳）有限公司 | Data transmission method and apparatus, computer device, and storage medium |
| CN110719245A (en) * | 2018-07-11 | 2020-01-21 | 中国移动通信集团浙江有限公司 | Internet of things data transmission method and system |
| CN110443695A (en) * | 2019-07-31 | 2019-11-12 | 中国工商银行股份有限公司 | Data processing method and its device, electronic equipment and medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860780A (en) * | 2021-03-30 | 2021-05-28 | 中国工商银行股份有限公司 | Data export method and device and terminal equipment |
| CN112860780B (en) * | 2021-03-30 | 2024-06-14 | 中国工商银行股份有限公司 | Data export method and device and terminal equipment |
| CN113283803B (en) * | 2021-06-17 | 2024-04-23 | 金蝶软件（中国）有限公司 | Method for making material demand plan, related device and storage medium |
| CN114679486A (en) * | 2022-03-29 | 2022-06-28 | 网易（杭州）网络有限公司 | Communication interaction method and system, application terminal, client terminal and transfer server terminal |
| CN114679486B (en) * | 2022-03-29 | 2024-05-07 | 网易（杭州）网络有限公司 | Communication interaction method and system, application terminal, client terminal and transfer server terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111510493B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111031058A (en) | Websocket-based distributed server cluster interaction method and device | |
| CN111510493B (en) | Distributed data transmission method and device | |
| CN110413673B (en) | Database data unified acquisition and distribution method and system | |
| CN113783922A (en) | Load balancing method, system and device | |
| CN112463535B (en) | Multi-cluster exception handling method and device | |
| CN111163130B (en) | Network service system and data transmission method thereof | |
| CN101815084A (en) | User-terminal file management system and application method thereof | |
| CN113259415B (en) | Network message processing method and device and network server | |
| CN113626002A (en) | Service execution method and device | |
| CN110764881A (en) | Distributed system background retry method and device | |
| CN111445331A (en) | Transaction matching method and device | |
| CN112953908A (en) | Network isolation configuration method, device and system | |
| CN105516086A (en) | Service processing method and apparatus | |
| CN113347257A (en) | Communication method, communication device, server and storage medium | |
| CN112689012A (en) | Cross-network proxy communication method and device | |
| CN112769945A (en) | Distributed service calling method and device | |
| CN112817694A (en) | Automatic load balancing method and device for distributed system | |
| CN112905338A (en) | Automatic allocation method and device for computing resources | |
| CN110427260B (en) | Host job scheduling method, device and system | |
| CN113326025B (en) | Single cluster remote continuous release method and device | |
| CN111352719A (en) | Transaction book-keeping service data processing method, device and system | |
| CN116069493A (en) | Data processing method, device, equipment and readable storage medium | |
| CN113645151A (en) | DUP equipment message management method and device | |
| CN110889765B (en) | Transaction information reporting method and device | |
| CN111782260A (en) | Gray scale distribution method and gray scale distribution device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |