CN118971954A - Data collaboration method based on QUIC protocol under multi-star weak network scene - Google Patents
Data collaboration method based on QUIC protocol under multi-star weak network scene Download PDFInfo
- Publication number
- CN118971954A CN118971954A CN202411457260.1A CN202411457260A CN118971954A CN 118971954 A CN118971954 A CN 118971954A CN 202411457260 A CN202411457260 A CN 202411457260A CN 118971954 A CN118971954 A CN 118971954A
- Authority
- CN
- China
- Prior art keywords
- file
- task
- quic
- transmission
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000005540 biological transmission Effects 0.000 claims abstract description 126
- 230000006854 communication Effects 0.000 claims abstract description 31
- 238000004891 communication Methods 0.000 claims abstract description 31
- 238000012795 verification Methods 0.000 claims abstract description 9
- 230000007246 mechanism Effects 0.000 claims description 15
- 238000007726 management method Methods 0.000 claims description 14
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 5
- 241000533950 Leucojum Species 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 230000002085 persistent effect Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 abstract description 4
- 230000003993 interaction Effects 0.000 abstract description 3
- 230000005012 migration Effects 0.000 abstract description 3
- 238000013508 migration Methods 0.000 abstract description 3
- 238000005516 engineering process Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 description 13
- 238000012546 transfer Methods 0.000 description 12
- 239000000872 buffer Substances 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Abstract
The invention discloses a data collaboration method under a multi-star weak network scene based on QUIC protocol; comprising the following steps: the QUIC-based HTTP service is used for file metadata management and transmission task management; socket assembly based on QUIC, providing two-way communication and file transmission functions; and the data collaboration service is used for realizing file synchronization, data migration and message communication. The invention realizes the efficient synchronization, breakpoint continuous transmission and safe integrity verification of the file through key technologies such as file differential management, task management, message communication module and the like, thereby ensuring the effective and stable data interaction capability between satellite nodes. The method is suitable for application scenes such as message communication, file synchronization, breakpoint continuous transmission and the like in the space-based network, and aims to improve the speed, reliability and safety of data transmission.
Description
Technical Field
The invention relates to the technical field of network communication, in particular to a data collaboration method under a multi-star weak network scene based on a QUIC protocol.
Background
The multi-satellite weak network scene refers to that in a constellation system formed by a plurality of satellites, a communication link of the multi-satellite weak network scene is influenced by factors such as the earth atmosphere, solar activity, satellite position change and the like, and the network condition is extremely unstable. The traditional TCP protocol is not good in performance under the conditions, and the QUIC (Quick UDP Internet Connections, fast UDP Internet connection) protocol is used as a novel transmission layer protocol based on UDP, has the advantages of fast connection establishment, low delay, multiplexing, high safety and the like, and is very suitable for being applied to the communication scene of a multi-star weak network. Therefore, the realization of a stable and reliable data cooperative system based on the QUIC protocol becomes a critical subject of multi-star task cooperation in a constellation.
The invention provides and realizes a data collaboration method under a multi-star weak network scene based on a QUIC protocol. The satellite network processes and stores large-scale data, the communication link of the satellite network is influenced by factors such as solar activity, satellite position change and the like, and the network condition is extremely unstable. The conventional TCP protocol performs poorly under these conditions, and connection jitter problems can occur. The QUIC protocol is used as a novel protocol realized based on UDP, has the advantages of quick connection establishment, low delay, multiplexing, high safety and the like, and is very suitable for being applied to satellite network communication. Therefore, the invention designs and realizes a message communication and data synchronization mechanism in a multi-star weak network environment based on the QUIC protocol.
Disclosure of Invention
The invention aims to provide a data collaboration method under a multi-star weak network scene based on a QUIC protocol, which provides simple and quick protocol coding and decoding, point-to-point file transmission capability, reliable breakpoint continuous transmission processing and safe integrity verification capability for satellite nodes, and ensures that each satellite node of a constellation system has effective information communication and data interaction capability in the weak network environment.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:
the invention provides a data collaboration method under a multi-star weak network scene based on a QUIC protocol, which comprises the following steps:
step 1: starting a QUIC service for file transmission and an http3 service based on the QUIC protocol on each satellite node;
Step 2: the satellite node requests file metadata of the target catalog from adjacent satellite nodes, performs differential comparison on the acquired file metadata and image files of the local catalog, and screens out a file list to be updated under the local catalog and a computing node where the file list is located;
Step 3: submitting the screened target file information to a task queue, and starting a file transmission task to monitor heartbeat;
step 4: establishing a QUIC client for file receiving, and establishing connection with a QUIC server on a satellite node where a target file is located; and sending task information of file transmission;
step 5: the QUIC client of the current node and the QUIC server of the target satellite node start to transmit file data; in the file data transmission process, the network between satellite nodes fluctuates; if the connection is not disconnected, triggering a retry mechanism based on the QUIC protocol, and attempting to resend the data; if the connection is disconnected, triggering a breakpoint continuous transmission mechanism;
Step 6: after the file transmission is completed, comparing the file with the received CRC check code; if the comparison is consistent, the original file is covered and the metadata of the file is synchronized; if the comparison is inconsistent, a transmission failure message is sent to wait for retransmission.
Further, the step 1 includes the following sub-steps:
(1.1) creating a QUIC server, and registering a first reader and a first writer for the server; the first writer realizes that the type of the incoming message of any type is encoded into a contracted protocol format, and magic numbers, version numbers, packet types and packet lengths are added as protocol heads; the first reader reads the data with the appointed packet length from the QUIC stream and decodes the data into a message object;
(1.2) registering with the QUIC service a first data stream processor and a message processor for file delivery.
Further, in the step 2, the satellite node periodically acquires file information of the neighboring nodes through a heartbeat mechanism.
Further, in the step 2, the obtained metadata of the files are subjected to differential comparison with the mirror image files of the local catalogue, the latest modified metadata information of each file is screened out, and the node where the file is located is found; if the file is a local node, the latest version of the file is local, and synchronization from other nodes is not needed; if the node is a non-native node, judging whether the node is already in a transmission task queue, and if the node is not in the transmission task queue, adding the node into the transmission task queue.
Further, the step 3 includes the following sub-steps:
(3.1) taking out a file task list to be synchronized from a task queue, generating a unique task id for each task by using a snowflake algorithm, and sending the unique task id to a node where a target file is located;
(3.2) after receiving the file transmission task, the target node submits the file transmission task to a sending queue, and stores the new transmission task into a local database;
(3.3) a file transmission task monitoring thread periodically sends heartbeat to a target node to acquire a file transmission task to be executed; after the task is acquired, the file receiving client is started.
Further, the step 4 includes the following sub-steps:
(4.1) creating a QUIC client, registering a second reader and a second writer for the client;
(4.2) registering a second data stream processor for file reception with the QUIC client;
(4.3) the QUIC client initiates a connection establishment request to the server, and after connection establishment, the data stream processors of the server and the client start to execute asynchronously, and wait for message communication and file transmission;
(4.4) the QUIC client sends information of the file transmission task to the server through the second writer, wherein the information comprises a transmission task unique id, a task type, a sender node IP, a receiver node IP, a task state, an absolute path of the file, a storage position of the receiver, whether the file needs compression or not, a file offset and a file total size; the client changes the task state into the ready state before sending the task information.
Further, the step 5 includes the following sub-steps:
(5.1) after the QUIC server reads the file transmission task sent by the client through the first reader, finding out the target file, if the file does not exist, sending a failure response to the QUIC client, and closing the connection;
(5.2) the server judges whether the task state is ready, if not, the server waits for the next message of the client; if the file is in a ready state, entering a file sending flow, and simultaneously starting an asynchronous thread to calculate a file CRC check code;
(5.3) the server side points the file pointer position to the offset position in the task information, and starts writing data into the QUIC stream by using a zero copy method; modifying the task state into incomplete state, and persisting the incomplete state into a local database;
(5.4) the client creates a temporary file for receiving file data while writing the data directly into the temporary file from the QUIC stream using a zero copy method;
(5.5) stopping writing after the client writes the data of the total size of the file and the offset of the file, and sending a message of completion of transmission to the server;
(5.6) the server receives the transmission completion message and sends a CRC check code to the client; after receiving the information, the client compares the information with the temporary file; if the verification is passed, the temporary file is named as a formal file name, and a successful transmission message is sent to the server; if the verification fails, deleting the temporary file, and sending a transmission failure message to the server;
(5.7) the server updates the task state according to the received transmission state message and persists to a local database; closing the connection;
(5.8) after the connection is disconnected, the server changes the task state into interrupt and stores the interrupt in a local database;
(5.9) obtaining the task in the interrupt state through the substep (3.3), and executing the step 4;
(5.10) in sub-step (4.4), modifying the file offset of the task metadata information to the total size of the local unfinished temporary file; and step 5, executing until the transmission task is executed.
The invention also provides a data collaboration system under the multi-star weak network scene based on the QUIC protocol, which comprises:
The file difference management module is used for providing a difference strategy of file catalogues among satellite nodes, screening out files of the latest version and updating the whole star group;
the file transmission task management module is used for managing file transmission tasks and is responsible for task submission, transmission state monitoring and flow control;
And the message communication module is used for realizing inter-satellite data communication service and file transmission service.
The invention also provides an electronic device comprising a memory and a processor, the memory being coupled to the processor; the storage is used for storing program data, and the processor is used for executing the program data to realize the data collaboration method under the QUIC protocol-based multi-star weak network scene.
The invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the above-mentioned data collaboration method in a multi-star weak network scenario based on the QUIC protocol.
The beneficial effects of the invention are as follows: the QUIC protocol used by the invention is used as a novel protocol realized based on UDP, has the advantages of quick connection establishment, low delay, multiplexing, high safety and the like, and is very suitable for being applied to satellite network communication. Therefore, the invention designs and realizes a message communication and data synchronization mechanism in a multi-star weak network environment based on the QUIC protocol. The invention realizes file metadata management and transmission task management based on the QUIC HTTP service; socket assembly based on QUIC, providing two-way communication and file transmission functions; and the data collaboration service is used for realizing file synchronization, data migration and message communication. The invention realizes the efficient synchronization, breakpoint continuous transmission and safe integrity verification of the file through key technologies such as file differential management, task management, message communication module and the like, thereby ensuring the effective and stable data interaction capability between satellite nodes.
Drawings
FIG. 1 is a diagram of a system architecture in the present invention;
FIG. 2 is a flow chart of the inter-satellite data synchronization in the present invention;
FIG. 3 is a file transfer data flow diagram in the present invention;
FIG. 4 is a breakpoint continuous flow chart in the present invention;
FIG. 5 is a transfer task state transition diagram in the present invention;
fig. 6 is a schematic diagram of an electronic device in the present invention.
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings. The features of the examples and embodiments described below may be combined with each other without conflict.
The general architecture of the data collaboration method under the multi-star weak network scene based on the QUIC protocol is shown in figure 1, and each satellite node deploys a data collaboration service (dataserver) for processing the decision of the synchronous file and the scheduling of the transmission task. Request-response communication is realized between data collaboration services on satellite nodes between different satellites through a QUIC-based HTTP SERVER for requesting file metadata information of a specified directory from a neighboring satellite node and submitting tasks of transmission files and acquisition of task states to a target satellite node. File transfer and message communication between dataserver of different nodes is implemented using a quinc-based socket component. When the system is started, each satellite node is started to be HTTP SERVER and socket server.
Referring to the data synchronization flow between satellite nodes of fig. 2, the method comprises the following steps:
Step 1: each satellite node initiates a QUIC service for file delivery and an http3 service based on the QUIC protocol.
(1.1) Creating a QUIC server, registering a Reader and Writer for the server. The Writer realizes that the type of the incoming message of any type is encoded into a contracted protocol format, and magic numbers, version numbers, packet types and packet lengths are added as protocol heads; the agreed protocol format is: the magic number is 1 byte, the version number is 4 bytes, the packet type is 1 byte, and the packet length is 4 bytes. The Reader reads the data of the specified packet length from the QUIC stream according to the agreed protocol format and decodes the data into a message object.
(1.2) Registering with the QUIC service a custom data stream processor (STREAM HANDLER) for file delivery, and a message processor (MESSAGE HANDLER) for incoming message processing and file processing.
It should be noted that, in the metadata acquisition stage, the data collaboration service running on the satellite node provides the file timing synchronization capability under the designated directory, and when the timing task is started, the file difference management module in the data collaboration service reads the metadata information of the file list under the local directory, and simultaneously sends the file metadata acquisition request to the neighboring node. The destination node traverses all files under the appointed directory, and returns the metadata information of the files to the source node. Wherein the file metadata is as shown in table 1: when the network is not communicated with the individual adjacent satellite nodes, the synchronization skips the nodes and waits for the next task. The current task enters the file comparison phase, step 2 is performed.
Table 1: file metadata parameter list
Metadata parameters | Description of the invention |
Filename | File name |
Path | Absolute path of directory where file is located |
NodeIp | Node ip where file is located |
CreateTime | File creation time |
ModTime | File modification time |
Step 2: the satellite node obtains all file metadata of the target catalog from the adjacent satellite nodes, and performs differential comparison on the obtained file metadata and the image files of the local catalog, and screens out a file list to be updated under the local catalog and a computing node where the file list is located.
(2.1) The satellite node periodically acquires file information of adjacent nodes through a heartbeat mechanism; the heartbeat is realized by sending file information requests to adjacent nodes every 1-30s (which can be written according to configuration and default is 5 s).
(2.2) The comparison policy of the local file and the remote file is: and screening out the latest file by comparing the latest file modification time.
In the comparison stage, the file difference management module executes a comparison strategy, performs difference comparison on all the obtained file metadata lists, screens out the latest modified metadata information of each file, and finds out the node where the file is located. If the file is a local node, the latest version of the file is local, and synchronization from other nodes is not needed; if the node is a non-native node, judging whether the node is already in a transmission task queue, if not, adding the node into the queue, and entering the step 3.
Step 3: submitting the screened target file information to a task queue; and simultaneously starting a file transmission task to monitor heartbeat.
(3.1) Taking out a file task list to be synchronized from a task queue, generating a unique task id for each task by using a snowflake algorithm, and sending the unique task id to a node where a target file is located;
(3.2) after receiving the file transmission task, the target node submits the file transmission task to a sending queue, and stores the new transmission task into a local Database (DB);
(3.3) a file transmission task monitoring thread sends heartbeats to a target node at regular time (every 1-30s; writing can be performed according to configuration, default is 5 s) to acquire a file transmission task to be executed; after the task is acquired, the file receiving client is started, and the step 4 is entered.
It should be noted that, in the task submitting stage, the metadata information of the file to be synchronized is taken out from the queue, and is encapsulated into a file transmission task object, the parameters of which are shown in table 2, and the task object is submitted to the target node through the HTTP service; after receiving the transmission task, the target node puts the transmission task into a task cache and persists the task into a local database, waits for the source node to initiate a synchronization request, and enters step 4.
Table 2: file transfer task metadata information
Transmitting task parameters | Parameter description |
Taskid | Task unique id, generated in step3 using snowflake algorithm |
TaskType | Task type, (upload-send file to target node/download-pull file from target node) |
NodeIp | Target satellite node ip |
State | File transfer task status |
Filename | Transmitting file names |
Savepath | Post-transfer file storage path |
FailNum | Number of transmission failures |
Offset | Offset, starting transmission from file offset |
Size | Transmitting total size of file-bytes |
Step 4: establishing a QUIC client for file receiving, and establishing connection with a QUIC server on a satellite node where a target file is located; and sending the task information of file transmission.
In the file transmission stage, a file sender is taken as a Server end (a Server end) of a socket, and a file receiver is taken as a Client end (a Client end) of the socket of the file. The transmission phase data flow diagram is shown in fig. 3, and the phase comprises the following core flows: task metadata synchronization process, file transmission process, breakpoint resume process (as in fig. 4), and termination transmission process.
(4.1) Creating a QUIC client, registering the same Reader and Writer for the client as in sub-step (1.1).
Task metadata synchronization process: the source node sends a task transmission request to the destination node at regular time, and the destination node reads the local database to acquire the file to be transmitted and the task which is not completed and returns the file to be transmitted. The source node obtains a file transmission task, and if the task state is [ BLOCK ]/[ UNFINISHED ] (the task state is shown in table 3, the state switching mechanism is shown in fig. 5), a substep (4.3) breakpoint continuous transmission flow is executed; if the task state is [ CANCEL ], executing a sub-step (4.4) transmission termination flow; if the task state is [ UNSTART ], the file transfer flow starts to be executed. Creating a Client cooperation of a socket component, realizing a QUIC stream processor and a message buffer (buffer) processor, initiating connection to a Server end of a destination node by a connection manager, and executing scheduling of the stream processor and the message processor; after connection is established, the Client creates a receiving directory, modifies the state of a transmission task to be READY, and sends task metadata to the Server. The process (step 4 in fig. 3) first uses the encoder of the socket component to encode the task metadata object into a binary stream using a default protocol format, writes the QUIC stream by the Writer of the socket component; and the Server reads data, reads the whole packet of data from the QUIC stream through a Reader of the socket component, and extracts metadata information of the transmission task by using a decoder.
Table 3: file transfer state description
Status parameters | Description of the invention |
UNSTART | After not starting-task commit, but socket connection is not established |
READY | Ready-QUIC connection establishment, client and Server preparation completion |
UNFINISHED | Incomplete-file transfer |
BLOCK | Break-connect disconnect |
FINISHED | Completion-completion of file transfer and CRC check pass |
FAILED | Failure-File transfer failure including task metadata synchronization failure, temporary File creation failure, write failure, QUIC stream read failure, CRC check failure, file modification time (modtime) update failure |
CANCEL | Transmission termination-stop file transmission task |
(4.2) Registering a custom data stream processor for file reception with the QUIC client.
The file transmission process comprises the following steps: the Server uses a zero copy mechanism to directly write the file content into the QUIC stream for transmission; the Client creates a temporary file and uses a zero copy mechanism to read the size-offset (total file size-file offset) size bytes to write the file from the QUIC stream.
And (4.3) the QUIC client initiates a connection establishment request to the server, and after connection establishment, the data stream processors of the server and the client start to execute asynchronously, and wait for message communication and file transmission.
Breakpoint continuous transmission process: in the file transmission process, when the connection detection mechanism of the QUIC protocol detects disconnection, the Server terminal updates the task state into BLOCK and stores the BLOCK into a local database. After the inter-satellite communication is recovered, the timing request in the substep (4.1) acquires the task information of the state, creates a Client, reads the temporary file size stored by the task, writes the temporary file size into task metadata as a file offset (offset), and sends the temporary file size to a Server. The metadata transmission procedure is the same as sub-step (4.1).
And (4.4) the QUIC client sends information of the file transmission task to the server through the Writer, wherein the information comprises a transmission task unique id, a task type, a sender node IP, a receiver node IP, a task state, an absolute path of the file, a storage position of the receiver, whether the file needs compression or not, a file offset and a file total size. The client changes the task state into the ready state before sending the task information.
Terminating the transmission process: the system receives the instruction of terminating a certain transmission task, sends the message to the Server, the Server closes the QUIC connection, and modifies the state of the transmission task to be [ CANCLE ]. And (3) the Client end cleans up the local temporary file after acquiring the terminated task from the timing task in the substep (4.1). After the cleaning is completed, a message is sent to the Server, and the task is cleaned from the local database.
Step 5: the QUIC client of the current node and the QUIC server of the target satellite node begin transmitting file data. In the file data transmission process, when the network between satellite nodes fluctuates; if the connection is not disconnected, triggering a retry mechanism based on the QUIC protocol, and attempting to resend the data; if the connection is disconnected, a breakpoint resume mechanism is triggered.
After the server reads the file transmission task sent by the client through the Reader, finding out a target file, if the file does not exist, sending a failure response to the QUIC client, and closing the connection;
(5.2) the server judges whether the task state is ready, if not, the server waits for the next message of the client; if the file is in a ready state, entering a file sending flow, and simultaneously starting an asynchronous thread to calculate a file CRC check code;
(5.3) the server side points the file pointer position to the offset position in the task information, and starts writing data into the QUIC stream by using a zero copy method; modifying the task state into incomplete state, and persisting the incomplete state into a local database;
(5.4) the client creates a temporary file for receiving file data while writing the data directly into the temporary file from the QUIC stream using a zero copy method;
(5.5) stopping writing after the client writes (total file size-file offset) data, and sending a transmission completion message to the server;
(5.6) the server receives the transmission completion message and sends a CRC check code to the client; after receiving the information, the client compares the information with the temporary file; checking, namely, more names of the temporary files are formal file names, and transmitting a successful transmission message to the server; the verification fails, the temporary file is deleted, and a transmission failure message is sent to the server;
(5.7) the server updates the task state according to the received transmission state message and persists to a local database; closing the connection;
(5.8) disconnecting, and changing the task state into interrupt by the server and storing the interrupt into a local database;
(5.9) obtaining the task of the interrupt state through the substep (3.3), and continuing to execute the step 4.
(5.10) In sub-step (4.4), modifying the file offset of the task metadata information to the total size of the local unfinished temporary file; and step 5, executing until the transmission task is executed.
It should be noted that, in the data verification stage and the transmission end stage, after the execution of the sub-step (4.2) is completed, the file is normally transmitted to the end. The Server calculates the CRC code of the file, codes the data and sends the data to the Client, and the coding transmission process is the same as the file transmission task information sending process in the substep (4.1). The Client calculates the temporary file stored locally and compares the temporary file with the received CRC check code to determine whether the temporary file is consistent with the received CRC check code. If the verification fails, changing the task state into [ FAILED ] and storing the [ FAILED ] into a local database; and (3) checking success, wherein the creation time (createtime) and the modification time (modtime) of the modified temporary file are consistent with the sender, the temporary file is renamed to be a formal file name, the original file is covered, and the task state is updated to be [ fixed ]. So far, when all file transmission is completed, the inter-satellite data synchronization task is finished.
Step 6: after the file transmission is completed, comparing the file with the received CRC check code; if the comparison is consistent, the original file is covered and the metadata of the file is synchronized; if the comparison is inconsistent, a transmission failure message is sent to wait for retransmission.
The following is an explanation of technical terms in the present invention:
QUIC-based HTTP service: for point-to-point between satellites, the request-response mode communication carrying a small amount of data provides a service interface, and the service realizes the functions of file metadata management and transmission task management across nodes.
A QUIC-based socket component for providing an abstract interface of the basic socket required for bi-directional communication of messages and file transfer.
Data collaboration service: for providing file synchronization, data migration, messaging services between the various satellite nodes.
Specifically, the socket assembly includes:
and the Server constructor is used for constructing the Server based on the QUIC protocol.
A Client constructor for constructing clients based on the QUIC protocol.
And the coder and the decoder are used for providing a default communication protocol format, comprising magic numbers, version numbers, packet types and packet lengths as protocol heads and encoding and decoding data according to the protocol.
Writer for writing data to the QUIC stream.
Reader for reading data from the QUIC stream.
The connection manager is an abstract process flow for providing a processing policy for connection establishment.
A QUIC stream processor, an abstract method, provided by the component user, is implemented at the Server or Client build time for implementing the method of operation on the QUIC stream in a concrete service.
A message buffer (buffer) processor, an abstract method provided by the component user, implemented in the Server or Client construction, is used for implementing the processing method of byte buffers generated by message sending and receiving in the service.
The invention provides a data collaboration system under a multi-star weak network scene based on QUIC protocol, which comprises:
And the file difference management module is used for providing a difference strategy of file catalogues among satellite nodes, and screening out files of the latest version for updating the whole satellite group.
And the file transmission task management module is used for managing file transmission tasks, and is responsible for task submission, transmission state monitoring and flow control.
And the message communication module is used for realizing inter-satellite data communication service and file transmission service based on the service provided by the socket component. The module realizes a QUIC stream processor and a message buffer (buffer) processor in a socket component, and is used for message communication under specific service scenes.
Corresponding to the foregoing embodiment of the data collaboration method in the multi-star weak network scenario based on the QUIC protocol, the embodiment of the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a data collaboration method in a multi-star weak network scenario based on the QUIC protocol as described above. As shown in fig. 6, a hardware structure diagram of an apparatus with data processing capability in any of the apparatuses with data processing capability in the multi-star weak network scenario based on the QUIC protocol according to the embodiment of the present application is shown in fig. 6, and in addition to the processor, the memory, the DMA controller, the magnetic disk, and the nonvolatile memory shown in fig. 6, any apparatus with data processing capability in the embodiment generally includes other hardware according to the actual function of the any apparatus with data processing capability, which is not described herein.
Corresponding to the foregoing embodiment of the data collaboration method in the multi-star weak network scenario based on the QUIC protocol, the embodiment of the present invention further provides a computer readable storage medium having a program stored thereon, where the program, when executed by a processor, implements the data collaboration method in the multi-star weak network scenario based on the QUIC protocol in the foregoing embodiment.
The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any of the data processing enabled devices described in any of the previous embodiments. The computer readable storage medium may also be any device having data processing capabilities, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), an SD card, a flash memory card (FLASH CARD), or the like, provided on the device. Further, the computer readable storage medium may include both internal storage units and external storage devices of any data processing device. The computer readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing apparatus, and may also be used for temporarily storing data that has been output or is to be output.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather to enable any modification, equivalent replacement, improvement or the like to be made within the spirit and principles of the invention.
The above embodiments are merely for illustrating the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement the same, the scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications according to the principles and design ideas of the present invention are within the scope of the present invention.
Claims (10)
1. The data collaboration method in the multi-star weak network scene based on the QUIC protocol is characterized by comprising the following steps:
step 1: starting a QUIC service for file transmission and an http3 service based on the QUIC protocol on each satellite node;
Step 2: the satellite node requests file metadata of the target catalog from adjacent satellite nodes, performs differential comparison on the acquired file metadata and image files of the local catalog, and screens out a file list to be updated under the local catalog and a computing node where the file list is located;
Step 3: submitting the screened target file information to a task queue, and starting a file transmission task to monitor heartbeat;
step 4: establishing a QUIC client for file receiving, and establishing connection with a QUIC server on a satellite node where a target file is located; and sending task information of file transmission;
step 5: the QUIC client of the current node and the QUIC server of the target satellite node start to transmit file data; in the file data transmission process, the network between satellite nodes fluctuates; if the connection is not disconnected, triggering a retry mechanism based on the QUIC protocol, and attempting to resend the data; if the connection is disconnected, triggering a breakpoint continuous transmission mechanism;
Step 6: after the file transmission is completed, comparing the file with the received CRC check code; if the comparison is consistent, the original file is covered and the metadata of the file is synchronized; if the comparison is inconsistent, a transmission failure message is sent to wait for retransmission.
2. The data collaboration method in a multi-star weak network scenario based on the QUIC protocol according to claim 1, wherein the step 1 comprises the following sub-steps:
(1.1) creating a QUIC server, and registering a first reader and a first writer for the server; the first writer realizes that the type of the incoming message of any type is encoded into a contracted protocol format, and magic numbers, version numbers, packet types and packet lengths are added as protocol heads; the first reader reads the data with the appointed packet length from the QUIC stream and decodes the data into a message object;
(1.2) registering with the QUIC service a first data stream processor and a message processor for file delivery.
3. The data collaboration method in a multi-star weak network scene based on the QUIC protocol according to claim 1, wherein in the step 2, the satellite node periodically acquires file information of the neighboring nodes through a heartbeat mechanism.
4. The method for data collaboration in a multi-star weak network scene based on the QUIC protocol according to claim 1, wherein in the step 2, the obtained metadata of the files are differentially compared with the mirror image files of the local directory, the latest modified metadata information of each file is screened out, and the node where the file is located is found; if the file is a local node, the latest version of the file is local, and synchronization from other nodes is not needed; if the node is a non-native node, judging whether the node is already in a transmission task queue, and if the node is not in the transmission task queue, adding the node into the transmission task queue.
5. The data collaboration method in a multi-star weak network scenario based on the QUIC protocol according to claim 2, wherein the step 3 comprises the following sub-steps:
(3.1) taking out a file task list to be synchronized from a task queue, generating a unique task id for each task by using a snowflake algorithm, and sending the unique task id to a node where a target file is located;
(3.2) after receiving the file transmission task, the target node submits the file transmission task to a sending queue, and stores the new transmission task into a local database;
(3.3) a file transmission task monitoring thread periodically sends heartbeat to a target node to acquire a file transmission task to be executed; after the task is acquired, the file receiving client is started.
6. The data collaboration method in a multi-star weak network scenario based on the QUIC protocol according to claim 5, wherein the step 4 comprises the following sub-steps:
(4.1) creating a QUIC client, registering a second reader and a second writer for the client;
(4.2) registering a second data stream processor for file reception with the QUIC client;
(4.3) the QUIC client initiates a connection establishment request to the server, and after connection establishment, the data stream processors of the server and the client start to execute asynchronously, and wait for message communication and file transmission;
(4.4) the QUIC client sends information of the file transmission task to the server through the second writer, wherein the information comprises a transmission task unique id, a task type, a sender node IP, a receiver node IP, a task state, an absolute path of the file, a storage position of the receiver, whether the file needs compression or not, a file offset and a file total size; the client changes the task state into the ready state before sending the task information.
7. The data collaboration method in a multi-star weak network scenario based on the QUIC protocol according to claim 6, wherein the step 5 comprises the following sub-steps:
(5.1) after the QUIC server reads the file transmission task sent by the client through the first reader, finding out the target file, if the file does not exist, sending a failure response to the QUIC client, and closing the connection;
(5.2) the server judges whether the task state is ready, if not, the server waits for the next message of the client; if the file is in a ready state, entering a file sending flow, and simultaneously starting an asynchronous thread to calculate a file CRC check code;
(5.3) the server side points the file pointer position to the offset position in the task information, and starts writing data into the QUIC stream by using a zero copy method; modifying the task state into incomplete state, and persisting the incomplete state into a local database;
(5.4) the client creates a temporary file for receiving file data while writing the data directly into the temporary file from the QUIC stream using a zero copy method;
(5.5) stopping writing after the client writes the data of the total size of the file and the offset of the file, and sending a message of completion of transmission to the server;
(5.6) the server receives the transmission completion message and sends a CRC check code to the client; after receiving the information, the client compares the information with the temporary file; if the verification is passed, the temporary file is named as a formal file name, and a successful transmission message is sent to the server; if the verification fails, deleting the temporary file, and sending a transmission failure message to the server;
(5.7) the server updates the task state according to the received transmission state message and persists to a local database; closing the connection;
(5.8) after the connection is disconnected, the server changes the task state into interrupt and stores the interrupt in a local database;
(5.9) obtaining the task in the interrupt state through the substep (3.3), and executing the step 4;
(5.10) in sub-step (4.4), modifying the file offset of the task metadata information to the total size of the local unfinished temporary file; and step 5, executing until the transmission task is executed.
8. A data collaboration system in a quinc protocol based multi-star weak network scenario implementing the method of claim 1, comprising:
The file difference management module is used for providing a difference strategy of file catalogues among satellite nodes, screening out files of the latest version and updating the whole star group;
the file transmission task management module is used for managing file transmission tasks and is responsible for task submission, transmission state monitoring and flow control;
And the message communication module is used for realizing inter-satellite data communication service and file transmission service.
9. An electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the data collaboration method in the quac protocol-based multi-star weak network scenario according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a data collaboration method in a quac protocol-based multi-star weak network scenario as claimed in any one of claims 1-7.
Publications (1)
Publication Number | Publication Date |
---|---|
CN118971954A true CN118971954A (en) | 2024-11-15 |
Family
ID=
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6061714A (en) | Persistent cache synchronization and start up system | |
EP0877320B1 (en) | Terminal emulator data stream differencing system | |
CA2205725C (en) | Preventing conflicts in distributed systems | |
US6453343B1 (en) | Methods, systems and computer program products for maintaining a common checkpoint cache for multiple sessions between a single client and server | |
EP1543420B1 (en) | Consistent message ordering for semi-active and passive replication | |
CN100369026C (en) | Transaction accelerator for client-server communication systems | |
US9286298B1 (en) | Methods for enhancing management of backup data sets and devices thereof | |
US8743881B2 (en) | Link data transmission method, node and system | |
US6335937B1 (en) | Node failure recovery in a hub and spoke data replication mechanism | |
EP1609063A1 (en) | State recovery and failover of intelligent network adapters | |
US7178060B2 (en) | Transmitting data reliably and efficiently | |
JP2002522845A (en) | Fault tolerant computer system | |
Liskov et al. | Efficient at-most-once messages based on synchronized clocks | |
WO2021147793A1 (en) | Data processing method, apparatus and system, electronic device and computer storage medium | |
US7159005B1 (en) | Methods, systems and computer program products for restartable multiplexed file transfers | |
CN118971954A (en) | Data collaboration method based on QUIC protocol under multi-star weak network scene | |
JP2000330777A (en) | Program exchanging method | |
JPH10320326A (en) | Check point communication processing system, method therefor and storage medium for storing the same method | |
US6237111B1 (en) | Method for logical connection resynchronization | |
Liskov et al. | Efficient at-most-once messages based on synchronized clocks | |
JP2776274B2 (en) | Virtual buffer control system in relay computer | |
KR100904085B1 (en) | Sequenced data transmission method | |
JPH10336242A (en) | Data communication system | |
CN112019360A (en) | Configuration data processing method, software defined network device, system and storage medium | |
CN117472642A (en) | High-reliability storage method of satellite network data based on coding |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication |