CN111510493B - Distributed data transmission method and device - Google Patents
Distributed data transmission method and device Download PDFInfo
- Publication number
- CN111510493B CN111510493B CN202010294993.3A CN202010294993A CN111510493B CN 111510493 B CN111510493 B CN 111510493B CN 202010294993 A CN202010294993 A CN 202010294993A CN 111510493 B CN111510493 B CN 111510493B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- target
- transfer processing
- processing server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a distributed data transmission method and a device, wherein the method comprises the following steps: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the capacity value of the target transmission data group and the authentication identification character string of the data source server; judging whether the sum of the residual spaces of all the transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking at least one transfer processing server as a target transfer processing server group, enabling all the transfer processing servers in the target transfer processing server group to respectively and parallelly receive all the target transmission data sent by the data source server, and carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain all file packets corresponding to the target transmission data; and respectively sending the file packets to the target data receiving servers. The application can improve the efficiency and reliability of data transmission on the basis of realizing parallel data transmission.
Description
Technical Field
The present application relates to the field of distributed architecture technologies, and in particular, to a distributed data transmission method and apparatus.
Background
In a distributed architecture, data is typically split into multiple small units and stored on different servers according to a distribution key. In disaster recovery scenarios, it is necessary to merge and backup the scattered data to a centralized backup server, or to distribute the data on the centralized backup server to a corresponding distributed server.
Existing financial systems backup or restore data on a source server through an interface. Under the distributed scene, the data cannot be monitored and processed in the transmission process, unified scheduling is lacking, the operations such as flow control, multi-server concurrent scheduling control, data conversion and data merging cannot be performed, once the backup process is started, intervention cannot be performed, the controllability and flexible diversity of the data backup or data recovery process are affected, and the reliability of data transmission is low.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a distributed data transmission method and a distributed data transmission device, which can improve the efficiency and the reliability of data transmission on the basis of realizing parallel data transmission.
In order to solve the technical problems, the application provides the following technical scheme:
in a first aspect, the present application provides a distributed data transmission method, including:
Receiving a data transmission request sent by a data source server, wherein the data transmission request comprises the following steps: a capacity value of a target transmission data group and an authentication identification character string of the data source server;
judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and respectively sending the file packages to a target data receiving server.
Further, before the receiving the data transmission request sent by the data source server, the method further comprises: receiving user authentication information sent by the data source server; and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
Further, after the file packets are respectively sent to the target data receiving servers, the method further comprises: and storing the target transmission data group to the target data receiving server.
Further, in the process that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the method further includes: and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, and if so, adjusting the flow to the flow threshold.
Further, the step of using the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, so that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, includes: and respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
In a second aspect, the present application provides a distributed data transmission apparatus, comprising:
the receiving request module is used for receiving a data transmission request sent by the data source server, wherein the data transmission request comprises the following components: a capacity value of a target transmission data group and an authentication identification character string of the data source server;
the processing module is used for judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as the target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data;
and the sending module is used for respectively sending the file packets to the target data receiving server.
Further, the distributed data transmission device further includes: the authentication information receiving module is used for receiving user authentication information sent by the data source server; and the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
Further, the distributed data transmission device further includes: and the storage module is used for storing the target transmission data group to the target data receiving server.
Further, in the process that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the processing module is further configured to determine whether the flow rate of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, and if yes, adjust the flow rate to the flow threshold.
Further, the processing module includes: and acquiring a target transmission data unit, wherein the target transmission data unit is used for respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the distributed data transmission method when executing the program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer instructions which, when executed, implement the distributed data transmission method.
As can be seen from the above technical schemes, the present application provides a distributed data transmission method and apparatus. Wherein the method comprises the following steps: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises the following steps: a capacity value of a target transmission data group and an authentication identification character string of the data source server; judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data; and respectively sending the file packages to a target data receiving server. The application can improve the efficiency and reliability of data transmission on the basis of realizing parallel data transmission; specifically, the data transmission process can be monitored in real time, such as flow monitoring and file content blacklist monitoring; the data is flexibly scheduled, parallel data transmission can be realized, the utilization rate of each server is improved, and the cost is saved; the reliability and the controllability of data transmission can be improved by splitting or merging the data, and the accuracy of determining the error node can be improved; the transfer processing server can be expanded, and different numbers of data interception processors can be expanded according to service requirements; the data transmission can be controlled in a customized and refined way, and the data flow passing through the transfer can be controlled according to a predefined method, such as flow control and file content blacklist control.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a distributed data transmission method according to an embodiment of the present application;
FIG. 2 is a flow chart of a distributed data transmission method according to another embodiment of the application;
fig. 3 is a schematic structural diagram of a distributed data transmission device according to an embodiment of the present application;
FIG. 4 is a topology diagram of a coordination server and a relay server for a specific application example of the present application;
FIG. 5 is a topology diagram of a data source server and a relay server of a specific application example of the present application;
FIG. 6 is a topology diagram of a relay server and a backup storage server for a specific application example of the present application;
fig. 7 is a schematic structural diagram of a transit server according to a specific application example of the present application;
FIG. 8 is a schematic diagram of a data source server according to an embodiment of the present application;
FIG. 9 is a data backup flow chart of an example of a specific application of the present application;
FIG. 10 is a data recovery flow chart of a specific application example of the present application;
fig. 11 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Based on this, in order to improve the efficiency and reliability of data transmission based on the realization of parallel data transmission, the embodiment of the application provides a distributed data transmission device, which may be a server or a client device, wherein the client device may include a smart phone, a tablet electronic device, a network set top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, an intelligent wearable device, and the like. Wherein, intelligent wearing equipment can include intelligent glasses, intelligent wrist-watch and intelligent bracelet etc..
In practical applications, the portion of the distributed data transmission may be performed on the server side as described above, or all operations may be performed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The application is not limited in this regard. If all operations are performed in the client device, the client device may further include a processor.
The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.
Any suitable network protocol may be used for communication between the server and the client device, including those not yet developed on the filing date of the present application. The network protocols may include, for example, TCP/IP protocol, UDP/IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol ), REST protocol (Representational State Transfer, representational state transfer protocol), etc. used above the above-described protocol.
The following examples are presented in detail.
In order to improve the efficiency and reliability of data transmission based on the realization of parallel data transmission, the present embodiment provides a distributed data transmission method in which an execution subject is a distributed data transmission device, wherein the distributed data transmission device may be a separate server; the transmission device can also be combined by the coordination server and the transfer processing server; at least one of the transfer processing servers may be further configured to store basic information, such as remaining space information and addresses, of each transfer processing server; as shown in fig. 1, the method specifically includes the following steps:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises: the capacity value of the target transmission data group and the authentication identification character string of the data source server.
Specifically, the authentication identification character string is a unique code of the data source server, and the data source server sending the data transmission request and the transfer processing server matched with the data source server can be determined according to the authentication identification character string. The target transmission data may be a financial transaction data file or a customer information file, etc.
S200: and judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data.
Specifically, the capacity value of each file package does not exceed the threshold value. The remaining space is the allocatable resource of the transfer processing server. The number of the transfer processing servers in the target transfer processing server group is the same as the number of the target transmission data. The threshold may be set according to the actual situation.
It can be understood that if the sum of the remaining spaces of the corresponding transfer processing servers is less than the capacity value of the target transmission data according to the authentication identification character string, a space shortage prompt message is sent to the data source server.
S300: and respectively sending the file packages to a target data receiving server.
Specifically, the data transmission method provided by the embodiment of the application is based on the distributed framework for data transmission.
In one embodiment, the determining, according to the authentication identifier string, whether the sum of the remaining spaces of the corresponding relay processing servers is greater than the capacity value of the target transmission data set includes:
and determining a transfer processing server matched with the data source server according to the authentication identification character string.
And judging whether the sum of the residual spaces of the transit processing servers is larger than the capacity value of the target transmission data group.
In one embodiment, the serving at least one corresponding relay processing server as the target relay processing server group according to the capacity value of the target transmission data group includes:
and sequencing the transfer processing servers according to a preset sequencing rule, and taking the transfer processing servers with the least number and the sum of the residual spaces obtained from front to back according to the sequencing result as the target transfer processing server group, wherein the sum of the residual spaces is larger than the capacity value.
Wherein the size of the target transmission data is related to the remaining space of each of the relay processing servers. The preset ordering rule can be set according to actual needs, and the application is not limited to this. For example, the ranking rule is that each relay processing server ranks from large to small according to respective remaining space or each relay processing server ranks from front to back according to respective start time. In one example, each transfer processing server with the remaining space is sorted from front to back according to the respective start time, and the minimum number of transfer processing servers with the sum of the remaining spaces greater than the capacity value of the target transmission data is obtained from front to back.
In one embodiment, before the splitting or merging process is performed on the target transmission data according to the same threshold, the method further includes: and labeling each target transmission data according to the authentication identification character string. And determining a corresponding data source server according to the target transmission data after labeling.
In order to further improve the reliability and security of data transmission, as shown in fig. 2, in an embodiment of the present application, before step S100, the method further includes:
s101: and receiving the user authentication information sent by the data source server.
S102: and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
Specifically, the distributed data transmission device performs identity verification on each to-be-matched transfer processing server according to the user authentication information, and if verification is passed, the corresponding to-be-matched transfer processing server is a transfer processing server matched with the data source server.
In order to implement parallel data backup, in one embodiment of the present application, after step S300, the method further includes:
And storing the target transmission data group to the target data receiving server.
In order to realize real-time monitoring of data transmission and improve reliability of data transmission, in one embodiment of the present application, in a process that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, the method further includes:
and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, and if so, adjusting the flow to the flow threshold.
The flow threshold may be set according to actual needs, which is not limited by the present application.
Specifically, the distributed data transmission device may further monitor in real time whether the target transmission data is in a preset file content blacklist, if yes, stopping the transmission process. In addition, the distributed data transmission device can also monitor the process of splitting or merging the target transmission data in real time, and if the processing fails, the processing process is restarted.
In order to further improve accuracy and efficiency of data transmission, in one embodiment of the present application, in step S200, the corresponding at least one relay processing server is used as a target relay processing server group according to the capacity value of the target transmission data group, so that each relay processing server in the target relay processing server group receives, respectively and in parallel, each target transmission data in the target transmission data group sent by the data source server, where the method includes:
S211: and respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
In order to improve the efficiency and reliability of data transmission based on parallel data transmission, the present application provides an embodiment of a distributed data transmission device for implementing all or part of the content in the distributed data transmission method, referring to fig. 3, where the distributed data transmission device specifically includes:
the receiving request module 10 is configured to receive a data transmission request sent by a data source server, where the data transmission request includes: the capacity value of the target transmission data group and the authentication identification character string of the data source server.
And the processing module 20 is configured to determine, according to the authentication identifier string, whether a sum of remaining spaces of the corresponding transfer processing servers is greater than a capacity value of the target transmission data set, if yes, use the corresponding at least one transfer processing server as a target transfer processing server set according to the capacity value of the target transmission data set, so that each transfer processing server in the target transfer processing server set receives, respectively and in parallel, each target transmission data in the target transmission data set sent by the data source server, and respectively performs splitting or merging processing on the target transmission data according to the same threshold value, to obtain each file packet corresponding to the target transmission data.
And the sending module 30 is used for respectively sending the file packets to the target data receiving servers.
In one embodiment of the present application, the distributed data transmission device further includes:
and the authentication information receiving module is used for receiving the user authentication information sent by the data source server.
And the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
In one embodiment of the present application, the distributed data transmission device further includes: and the storage module is used for storing the target transmission data group to the target data receiving server.
In one embodiment of the present application, in a process that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server, respectively and in parallel, the processing module is further configured to determine whether a flow rate of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, and if yes, adjust the flow rate to the flow threshold.
In one embodiment of the present application, the processing module includes:
and acquiring a target transmission data unit, wherein the target transmission data unit is used for respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
The embodiments of the distributed data transmission apparatus provided in the present disclosure may be specifically used to execute the processing flow of the embodiments of the distributed data transmission method, and the functions thereof are not described herein again, and reference may be made to the detailed description of the embodiments of the distributed data transmission method.
In order to improve the data scheduling, controlling, monitoring and processing capability in the distributed data transmission and further improve the data transmission efficiency and reliability, the application also provides a specific application example of the distributed data transmission system, wherein in the specific application example, the specific distributed data transmission system consists of a coordination server 1, a transit server 2, a data source server 3 and a backup storage server 4. The relay server 2 is configured to implement the functions of the relay processing server and/or the distributed data transmission device, and the backup storage server 4 is configured to implement the functions of the target data receiving server, which is specifically described as follows:
(1) As shown in fig. 4, the coordination server 1 is a control device for coordinating and controlling the whole backup transfer process, and includes two servers with high availability; the coordination server comprises a high availability module 101 and a communication module 102.
A high availability module 101, configured to, when a primary server is available, make the primary server an active server and make the backup server inactive; when the main server is down, the backup server is an active server, and the management rights of all transfer servers are handed over to the existing active server.
The communication module 102 is configured to coordinate network communication between the server and the transit server 2.
(2) As shown in fig. 5 to fig. 7, the relay server 2 is a carrying device for data transmission processing, and includes a data pulling module 201, a data pushing module 202, a data backup server module 301 and a relay function module 302.
The data pulling module 201 is configured to pull data to be backed up from a plurality of data source servers according to the real-time load.
The data pushing module 202 is configured to push the processed backup data to the backup storage server for storage according to the data processing progress, or restore the backup data to the relay server for processing.
The data backup server module 301 is configured to perform a data processing process.
The transfer function module 302 includes a data receiver, a data monitor, a data processor, and a data transmitter. The data receiver is used for receiving data sent by the data source server or the backup storage server, the data monitor is used for monitoring and managing data streams, the data processor is used for converting the data, and the data transmitter is used for transmitting the data to the backup storage data or the data source server.
(3) Referring to fig. 8, the data source server 3 is a server storing data to be backed up, and includes a data backup client module 401 and a data transmitting and receiving module 402.
The data backup client module 401 is configured to send and receive data.
The data sending and receiving module 402 includes a data sender and a data receiver, which are respectively used for sending data to and receiving data from a data backup server on the transfer server.
(4) The backup storage server 4 is used to store data.
As can be seen from the above description, in this specific application example, the interceptor is extensible; the transmission data can be finely controlled, and the data flow passing through the transfer can be controlled according to a predefined method, such as flow control and file content blacklist control; the size of the transfer cluster can be transversely expanded and reduced, and the transfer cluster can be expanded and reduced according to the sizes of the data source server and the backup storage server, so that resources are reasonably used.
In order to further explain the scheme, in combination with the distributed data transmission system in the above specific application example, the present application further provides a specific application example of a distributed data backup method, as shown in fig. 9, which specifically includes the following contents:
Step 501: the backup client performs identity authentication, after the identity authentication passes, the backup server returns an authentication identification character string, and sets effective time, and a subsequent request of the backup client needs to carry the authentication identification character string as a request parameter.
Specifically, the backup client initiates an identity authentication request to the backup server. The valid time is a valid time set in the relay server. The authentication identification character string is used for authenticating and identifying the backup client by the backup server.
Step 502: the backup client applies resources to the backup server, the backup server allocates the resources according to the current resources, and if the application resources of the backup client are larger than the highest allocable resources, the resources are returned to be insufficient; otherwise, returning the resource detail allocated to the backup client. The backup client may send the file to the specified directory returned by the server.
Specifically, the resource details include available transit servers, available catalogs and effective time which are obtained by application distribution. And the effective time represents the maximum value of the time consumed by the backup server to send the resource detail to receive the file sent by the backup client, and if the consumed time exceeds the effective time, the corresponding data transmission task is ended.
Step 503: and the backup client sends the file to the applied transfer server.
Step 504: the backup server intercepts the file, processes the file according to the functions on the processing chain and monitors the file in real time. The backup server side intercepts a file being sent by the backup client side and comprises file format conversion and file splitting and merging, wherein the file format conversion is that the file is marked with an invisible label for file identification, the file splitting operation merges small files into 300G files, and the large files are split into a plurality of 300G files.
Step 505: and the backup server sends the files after the processing to the backup storage server for storage.
Step 506: the backup server side cleans up the temporary file generated by the backup process.
The temporary file is an intermediate file generated in the interception process and a file which is successfully stored in the backup storage server and is sent by the backup client is received by the transfer server. Such as log files.
In order to further explain the scheme, in combination with the distributed data transmission system in the above specific application example, the present application further provides a specific application example of the distributed data recovery method, as shown in fig. 10, which specifically includes the following contents:
Step 601: the backup client performs identity authentication, after the identity authentication passes, the backup server returns an authentication identification character string, and sets effective time, and a subsequent request of a caller needs to carry the authentication identification character string as a request parameter.
Step 602: the backup client applies resources to the backup server, the backup server allocates the resources according to the current resources, and if the application resources of the backup client are larger than the highest allocable resources, the resources are returned to be insufficient; otherwise, returning the resource detail allocated to the backup client. The backup client may send the file to the specified directory returned by the server.
Step 603: the backup server side requests the file from the backup storage server and receives the file to the appointed directory.
Step 604: the backup server intercepts the file, processes the file according to the function on the processing chain and monitors the file.
Step 605: and the backup server sends the processed file to the backup client for recovery.
Step 606: the backup server side cleans up the temporary file generated by the recovery processing.
As can be seen from the above description, the distributed data transmission method and apparatus provided by the present application can improve the efficiency and reliability of data transmission based on parallel data transmission; specifically, the application can improve the capability of data state monitoring, data conversion and data scheduling in the data transmission process, can improve the data processing efficiency by using the distributed architecture, can realize the data backup in the scattered data set to the intensive backup storage server, and ensures the consistency of the data.
In order to improve the efficiency and reliability of data transmission based on parallel data transmission, the application provides an embodiment of an electronic device for implementing all or part of the contents in the distributed data transmission method, which specifically comprises the following contents:
a processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between the distributed data transmission device and related equipment such as a user terminal; the electronic device may be a desktop computer, a tablet computer, a mobile terminal, etc., and the embodiment is not limited thereto. In this embodiment, the electronic device may be implemented with reference to an embodiment for implementing the distributed data transmission method and an embodiment for implementing the distributed data transmission apparatus, and the contents thereof are incorporated herein, and are not repeated here.
Fig. 11 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 11, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 11 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.
In one or more embodiments of the application, the distributed data transfer functionality may be integrated into the central processor 9100. The central processor 9100 may be configured to perform the following control:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises the following steps: the capacity value of the target transmission data group and the authentication identification character string of the data source server.
S200: and judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data.
S300: and respectively sending the file packages to a target data receiving server.
As can be seen from the above description, the electronic device provided by the embodiment of the present application can improve the efficiency and reliability of data transmission on the basis of implementing parallel data transmission.
In another embodiment, the distributed data transmission apparatus may be configured separately from the central processor 9100, for example, the distributed data transmission apparatus may be configured as a chip connected to the central processor 9100, and the distributed data transmission function is implemented by control of the central processor.
As shown in fig. 11, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 11; in addition, the electronic device 9600 may further include components not shown in fig. 11, and reference may be made to the related art.
As shown in fig. 11, the central processor 9100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.
The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.
The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.
The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.
As can be seen from the above description, the electronic device provided by the embodiment of the present application can improve the efficiency and reliability of data transmission on the basis of implementing parallel data transmission.
The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the distributed data transmission method in the above embodiments, the computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements all the steps of the distributed data transmission method in the above embodiments, for example, the processor implements the steps of:
s100: receiving a data transmission request sent by a data source server, wherein the data transmission request comprises the following steps: the capacity value of the target transmission data group and the authentication identification character string of the data source server.
S200: and judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data.
S300: and respectively sending the file packages to a target data receiving server.
As can be seen from the above description, the computer readable storage medium provided by the embodiments of the present application can improve the efficiency and reliability of data transmission based on parallel data transmission.
The embodiments of the method of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment mainly describes differences from other embodiments. For relevance, see the description of the method embodiments.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principles and embodiments of the present application have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (10)
1. A distributed data transmission method, comprising:
receiving a data transmission request sent by a data source server, wherein the data transmission request comprises the following steps: a capacity value of a target transmission data group and an authentication identification character string of the data source server;
judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking at least one corresponding transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, and respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data, and if not, sending insufficient space prompt information to the data source server;
respectively sending the file packages to a target data receiving server;
the method uses at least one corresponding transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, and comprises the following steps:
Sequencing the transfer processing servers according to a preset sequencing rule, and taking the transfer processing servers with the least number and the sum of the residual spaces obtained from front to back according to the sequencing result as the target transfer processing server group, wherein the sum of the residual spaces is larger than the capacity value;
the step of using at least one corresponding transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, so that each transfer processing server in the target transfer processing server group receives each target transmission data in the target transmission data group sent by the data source server respectively and in parallel, including:
and respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
2. The distributed data transmission method according to claim 1, further comprising, prior to said receiving the data transmission request sent by the data source server:
receiving user authentication information sent by the data source server;
and determining each transfer processing server matched with the data source server according to the user authentication information, and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
3. The distributed data transmission method according to claim 1, further comprising, after said transmitting the file packets to the target data receiving servers, respectively:
and storing the target transmission data group to the target data receiving server.
4. The distributed data transmission method according to claim 1, wherein in the process of receiving, by each relay processing server in the target relay processing server group, each target transmission data in the target transmission data group sent by the data source server, respectively and in parallel, further comprising:
and judging whether the flow of the target transmission data sent to the corresponding transfer processing server exceeds a flow threshold, and if so, adjusting the flow to the flow threshold.
5. A distributed data transmission apparatus, comprising:
the receiving request module is used for receiving a data transmission request sent by the data source server, wherein the data transmission request comprises the following components: a capacity value of a target transmission data group and an authentication identification character string of the data source server;
the processing module is used for judging whether the sum of the residual spaces of the corresponding transfer processing servers is larger than the capacity value of the target transmission data group according to the authentication identification character string, if so, taking the corresponding at least one transfer processing server as a target transfer processing server group according to the capacity value of the target transmission data group, enabling each transfer processing server in the target transfer processing server group to respectively and parallelly receive each target transmission data in the target transmission data group sent by the data source server, respectively carrying out segmentation or merging processing on the target transmission data according to the same threshold value to obtain each file packet corresponding to the target transmission data, and if not, sending insufficient space prompt information to the data source server;
the sending module is used for respectively sending the file packets to a target data receiving server;
The processing module comprises: the sequencing unit is used for sequencing the transfer processing servers according to a preset sequencing rule, and taking the transfer processing servers with the least number and the sum of the residual spaces obtained from front to back according to the sequencing result as the target transfer processing server group, wherein the sum of the residual spaces is larger than the capacity value;
the processing module comprises:
and acquiring a target transmission data unit, wherein the target transmission data unit is used for respectively sending the address, the corresponding storage catalogue and the residual space information of each transfer processing server in the target transfer processing server group to the data source server, so that the data source server splits the target transmission data group according to the residual space information of each transfer processing server to obtain at least one target transmission data, and respectively and parallelly sending the target transmission data to each transfer processing server according to the address and the corresponding storage catalogue of each transfer processing server.
6. The distributed data transmission apparatus of claim 5, further comprising:
the authentication information receiving module is used for receiving user authentication information sent by the data source server;
And the authentication module is used for determining each transfer processing server matched with the data source server according to the user authentication information and returning an authentication identification character string corresponding to the transfer processing server to the data source server.
7. The distributed data transmission apparatus of claim 5, further comprising:
and the storage module is used for storing the target transmission data group to the target data receiving server.
8. The distributed data transmission apparatus according to claim 5, wherein, in the process of receiving, by each relay processing server in the target relay processing server group, each target transmission data in the target transmission data group sent by the data source server, respectively and in parallel, the processing module is further configured to determine whether a traffic of the target transmission data sent to the corresponding relay processing server exceeds a traffic threshold, and if yes, adjust the traffic to the traffic threshold.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the distributed data transmission method of any of claims 1 to 4 when executing the program.
10. A computer readable storage medium having stored thereon computer instructions, which when executed implement the distributed data transmission method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010294993.3A CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010294993.3A CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111510493A CN111510493A (en) | 2020-08-07 |
| CN111510493B true CN111510493B (en) | 2023-09-26 |
Family
ID=71876136
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010294993.3A Active CN111510493B (en) | 2020-04-15 | 2020-04-15 | Distributed data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111510493B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112860780B (en) * | 2021-03-30 | 2024-06-14 | 中国工商银行股份有限公司 | Data export method and device and terminal equipment |
| CN113283803B (en) * | 2021-06-17 | 2024-04-23 | 金蝶软件（中国）有限公司 | Method for making material demand plan, related device and storage medium |
| CN114679486B (en) * | 2022-03-29 | 2024-05-07 | 网易（杭州）网络有限公司 | Communication interaction method and system, application terminal, client terminal and transfer server terminal |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911772A (en) * | 2017-02-20 | 2017-06-30 | 联想(北京)有限公司 | Server-assignment method, server-assignment device and electronic equipment |
| WO2019075978A1 (en) * | 2017-10-16 | 2019-04-25 | 平安科技（深圳）有限公司 | Data transmission method and apparatus, computer device, and storage medium |
| CN110443695A (en) * | 2019-07-31 | 2019-11-12 | 中国工商银行股份有限公司 | Data processing method and its device, electronic equipment and medium |
| CN110719245A (en) * | 2018-07-11 | 2020-01-21 | 中国移动通信集团浙江有限公司 | Internet of things data transmission method and system |
-
2020
- 2020-04-15 CN CN202010294993.3A patent/CN111510493B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106911772A (en) * | 2017-02-20 | 2017-06-30 | 联想(北京)有限公司 | Server-assignment method, server-assignment device and electronic equipment |
| WO2019075978A1 (en) * | 2017-10-16 | 2019-04-25 | 平安科技（深圳）有限公司 | Data transmission method and apparatus, computer device, and storage medium |
| CN110719245A (en) * | 2018-07-11 | 2020-01-21 | 中国移动通信集团浙江有限公司 | Internet of things data transmission method and system |
| CN110443695A (en) * | 2019-07-31 | 2019-11-12 | 中国工商银行股份有限公司 | Data processing method and its device, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111510493A (en) | 2020-08-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111510493B (en) | Distributed data transmission method and device | |
| CN111031058A (en) | Websocket-based distributed server cluster interaction method and device | |
| CN110958281B (en) | Data transmission method and communication device based on Internet of things | |
| CN114172662B (en) | Block chain external data acquisition method and device | |
| CN111163130B (en) | Network service system and data transmission method thereof | |
| CN112463535B (en) | Multi-cluster exception handling method and device | |
| CN101815084A (en) | User-terminal file management system and application method thereof | |
| CN111275555B (en) | Block chain transaction processing method, transaction node and block chain system | |
| CN112905338B (en) | Automatic computing resource allocation method and device | |
| CN113259415B (en) | Network message processing method and device and network server | |
| CN113626002A (en) | Service execution method and device | |
| CN110764881A (en) | Distributed system background retry method and device | |
| CN111445331A (en) | Transaction matching method and device | |
| CN110675159A (en) | Financial market transaction advance risk control method and system and electronic equipment | |
| CN105516086A (en) | Service processing method and apparatus | |
| CN112953908A (en) | Network isolation configuration method, device and system | |
| CN112689012A (en) | Cross-network proxy communication method and device | |
| CN113347257A (en) | Communication method, communication device, server and storage medium | |
| CN111352719B (en) | Transaction bookkeeping service data processing method, device and system | |
| CN112769945A (en) | Distributed service calling method and device | |
| CN113326025B (en) | Single cluster remote continuous release method and device | |
| CN116069493A (en) | Data processing method, device, equipment and readable storage medium | |
| CN112734545A (en) | Block chain data sharing method, device and system | |
| CN111782366A (en) | Distributed task scheduling method and device | |
| CN111782260A (en) | Gray scale distribution method and gray scale distribution device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |