CN108881400B - A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards - Google Patents
A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards Download PDFInfo
- Publication number
- CN108881400B CN108881400B CN201810521643.9A CN201810521643A CN108881400B CN 108881400 B CN108881400 B CN 108881400B CN 201810521643 A CN201810521643 A CN 201810521643A CN 108881400 B CN108881400 B CN 108881400B
- Authority
- CN
- China
- Prior art keywords
- file
- channel
- uploaded
- network interface
- control module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
- H04L69/162—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields involving adaptations of sockets based mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to field of communication technology, in particular to a kind of system for realizing that multi-channel data uploads in real time based on more network interface cards.Using physical machine as more network interface card systems of carrier, which is corresponding with multiple uploading channels；Locally file to be uploaded and verification is localized by top control module automatic identification；The top control module specifies uploading channel to distribute upload task using discrete logarithm automatic feeding；Each channel obtains the list of configuration file middle port, while obtaining this channel in memory and needing the task list completed；Then certain thread is generated according to the number of concurrent of configuration；File is uploaded into cloud after verifying by cloud.The present invention reaches timely, efficient, the stable transmission of data, meets multiple business application, piecemeal multi-thread transmission mechanism improves efficiency of transmission, is classified the system functions such as fault diagnosis.
Description
(1) technical field
It is the present invention relates to field of communication technology, in particular to a kind of to realize what multi-channel data uploaded in real time based on more network interface cards
System.
(2) background technique
In all trades and professions, database file, image are shared by more and more integrated application system institutes between text file
It uses, such as Real-time Monitoring Data, database data, small image.These data can be by way of internet, wifi
Internet is accessed, realization is interconnected, and then is realized shared.But in some industries, the generation of data is to be interconnected net isolation
, not good method data in real time, safety, be efficiently transmitted to application system, reduce the production efficiency of enterprise.
(3) summary of the invention
In order to compensate for the shortcomings of the prior art, the present invention provides one kind based on more network interface cards realize multi-channel data it is real-time on
The system of biography.
The present invention is achieved through the following technical solutions:
A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that:
Using physical machine as more network interface card systems of carrier, which is corresponding with multiple uploading channels；
Locally file to be uploaded and verification is localized by top control module automatic identification；The top control module use from
Dissipating algorithm automatic feeding specifies uploading channel to distribute upload task；
Each channel obtains the list of configuration file middle port, while obtaining this channel in memory and needing completing for task
List；Then certain thread is generated according to the number of concurrent of configuration；File is uploaded into cloud after verifying by cloud.
Wherein, the file that the top control module identifies is compared judgement with local metadata, if not there is upper biography
Record, is loaded into memory file；It records, is then directly recorded in local data base if there is uploading.
Further, the executive program finds that the file to be uploaded under specified file catalogue, file pass through local school automatically
It tests interface and judges whether file has uploaded, the local verification of upper transmitting file is by the way of MD5；Executive program passes through discrete calculation
Method is distributed file to be uploaded automatically and is uploaded to specified uploading channel.
Further, if by judgement verification, the file information is automatically loaded in memory, after file uploads in memory,
It can delete from memory, then be logged into local data base.
Further, the single channel uploads by way of concurrent port specifically: it is that the single channel is supported and
Hair number can carry out visual configuration.
Further, the amount of survival of port thread, dynamic generation thread can be identified in the single channel upload procedure
The thread that polishing is withered away；The generation of thread is generated by socket bundling port.
Further, the cloud verification specifically: the upload task in the single channel can be judged by way of MD5
Whether cloud file is stored, if it does, the upload task in channel discharges automatically, space in releasing memory first, with
After local data base is recorded.
Further, more network interface card systems have the function of bond.
The present invention is a set of safe and reliable efficient mass small documents data real-time radio Transmission system.System can be real-time
The mass small documents data that arbitrary system high speed generates are monitored to realize using the concurrent joint-action mechanism of more network interface cards and double verification schemes
All data are accurately uploaded to cloud in real time by Multi-channel multi-line journey dynamic load upload function.Its principle are as follows:
Server realizes multichannel by configuring more network interface cards, and file data to be uploaded is distributed to by executive program
Multiple upload subprocess.It is distributed to the file of subprocess and uploads metadata management and carry out upload verification, verify after passing through again
Check whether cloud needs to upload.
Uploading bandwidth is made full use of by configuring the concurrent quantity of upload, concurrently uploads and is realized using the mode of upwards transmission port
Single channel multithreading uploads in real time, and multichannel, which uploads, to carry out under the different environment such as city, rural area, tunnel simultaneously
It passes.Each channel has independent data-handling capacity, uploads the data finished and local data base is recorded, compared to traditional
Single-point, which uploads, provides better scalability and fault-tolerance.
The beneficial effects of the present invention are: the present invention generates grass-roots unit's small documents for existing system production process, it is mostly objective
Family end, the application demand of safe and efficient frequent transmission design a kind of long distance high efficiency document transmission system based on Transmission Control Protocol, reach
Transmission timely to data, efficient, stable, monitoring triggering upload mechanism, one of the main realization based on file, document creation event
Secondary to shake hands, lifelong transmission mechanism, customized distribution mechanisms meet multiple business application, piecemeal multi-thread transmission mechanism is improved and passed
The system functions such as defeated efficiency, classification fault diagnosis.
(4) attached drawing
Fig. 1: executive program flow chart；
Fig. 2: single channel, which uploads, passes through concurrent port flow figure.
(5) specific embodiment.
Embodiment
The present invention is more network interface card systems using physical machine as carrier, which is corresponding with multiple uploading channels；Object
Reason machine can be specially server, support more network card configurations, and network interface card support is extending transversely, does not have the upper limit in software view.Wherein
Top control module is responsible for the automatic discovery of upper transmitting file to be done, localization automatic Verification, and is referred to by discrete logarithm automatic feeding
Determine uploading channel distribution upload task.It supports network interface card dynamic configuration, environment hardware configuration can be uploaded according to practical, increasing adds deduct
Few uploading channel；And it can support the bond function of network interface card, eliminate because the damage of Single NIC causes single channel not available
Situation.
Top control module specifies uploading channel to distribute upload task by discrete logarithm automatic feeding, specifically:
There is specified catalogue to be uploaded on S11 local data base, be stored with file to be uploaded under catalogue to be uploaded；
The file increased newly under S12 top control module automatic identification catalogue to be uploaded, identification range includes subdirectory and specific item
Record；
The file that S13 executive program identifies is compared judgement with local metadata, and file passes through local verification interface
Judge whether file has uploaded, the local verification of upper transmitting file is by the way of MD5；If not thering is upload to record, file
It is loaded into memory；It is recorded if there is uploading, then directly records local data base；
S14 executive program distributes file to be uploaded by discrete logarithm automatically and is uploaded to specified uploading channel.
When file to be uploaded enters single channel (the hereinafter referred to as single channel), single channel obtains configuration file middle-end
Mouth list, while obtaining this channel in memory and needing the task list completed；Then it is generated centainly according to the number of concurrent of configuration
Thread；File is uploaded into cloud after verifying by cloud.Wherein, single channel, which uploads, supports to be accounted for according to the port under physical environment
Available port is configured with situation, supports the cloud school that is concurrent again, supporting transmitting file on single channel for uploading thread in single channel
It tests, a variety of dimensional informations of transmitting file on record support.
Specific steps are as follows:
S21 obtains the list of configuration file middle port, while obtaining this channel in memory and needing the task list completed；
S22 checks whether port configuration is correct, and task whether there is；Judge whether port can be used, it is such as available, dynamically with
Uploading channel is established in distal end.If it is not, directly abandoning local terminal slogan, and it is marked；It is generated according to the number of concurrent of configuration
Certain thread；Task, which exists, then continues upload task, and there is no then continue waiting for task；
Specifically: single channel obtains the concurrent configuration information in port, is judged；The correct then dynamic generation thread pool of configuration,
Configuration error then exits；Single channel obtains this channel task of memory, is judged；Task, which exists, then generates task list, task
There is no then continue to wait for；In the presence of port configuration is correct and task, the thread of configuration obtains task execution and uploads task；
The thread that S23 previous step generates can select file to be uploaded to carry out uploading to cloud from memory.Before uploading first
Cloud verification is carried out, has such as been uploaded, is directly abandoned, and removed from memory, and local data base is written；If do not uploaded, directly
Cloud is uploaded to, and is removed from memory, and local data base is written；
Cloud verification specifically:
Upload task in single channel can judge whether cloud file is stored by way of MD5, if it does, logical
Upload task in road discharges automatically, first space in releasing memory, and local data base is then recorded.
The present invention is described by way of example above, but the present invention is not limited to above-mentioned specific embodiment, all to be based on
Any changes or modifications that the present invention is done are fallen within the scope of the claimed invention.
Claims (8)
1. a kind of system for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: using physical machine as carrier
More network interface card systems, more network interface card systems are corresponding with multiple uploading channels；
Locally file to be uploaded and verification is localized by top control module automatic identification；The top control module uses discrete calculation
Method automatic feeding specifies uploading channel to distribute upload task；
Each channel obtains the list of configuration file middle port, while obtaining the task column that this channel needs to complete in memory
Table；Then certain thread is generated according to the number of concurrent of configuration；File is uploaded into cloud after verifying by cloud；
Top control module specifies uploading channel to distribute upload task by discrete logarithm automatic feeding, specifically:
There is specified catalogue to be uploaded on S11 local data base, be stored with file to be uploaded under catalogue to be uploaded；
The file increased newly under S12 top control module automatic identification catalogue to be uploaded, identification range includes subdirectory and sub- subdirectory；
The file that S13 top control module identifies is compared judgement with local metadata, and file passes through local verification interface judgement
Whether file has uploaded, and the local verification of upper transmitting file is by the way of MD5；If not there is upload to record, file is loaded
Into memory；It is recorded if there is uploading, then directly records local data base；
S14 top control module distributes file to be uploaded by discrete logarithm automatically and is uploaded to specified uploading channel.
2. the system according to claim 1 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: institute
It states the file that top control module identifies and is compared judgement with local metadata, if not there is upload to record, file is loaded into
In memory；It records, is then directly recorded in local data base if there is uploading.
3. the system according to claim 1 or 2 for being realized that multi-channel data uploads in real time based on more network interface cards, feature are existed
In: the top control module finds that the file to be uploaded under specified file catalogue, file pass through local verification interface judgement text automatically
Whether part has uploaded, and the local verification of upper transmitting file is by the way of MD5；Top control module by discrete logarithm distribute automatically to
Upper transmitting file is uploaded to specified uploading channel.
4. the system according to claim 3 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: such as
Fruit is automatically loaded in memory by judgement verification, the file information, after file uploads in memory, can be deleted from memory, so
After be logged into local data base.
5. the system according to claim 1 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: single
The number of concurrent that a channel is supported can carry out visual configuration.
6. the system according to claim 1 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: single
The amount of survival of port thread, the thread that dynamic generation thread polishing is withered away can be identified in a channel upload procedure；Line
The generation of journey is generated by socket bundling port.
7. the system according to claim 1 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: institute
State cloud verification specifically: whether the upload task in the single channel can judge cloud file by way of MD5
Storage, if it does, the upload task in channel discharges automatically, local data is then recorded in space in releasing memory first
Library.
8. the system according to claim 1 for realizing that multi-channel data uploads in real time based on more network interface cards, it is characterised in that: institute
Stating more network interface card systems has the function of bond.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810521643.9A CN108881400B (en) | 2018-05-28 | 2018-05-28 | A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810521643.9A CN108881400B (en) | 2018-05-28 | 2018-05-28 | A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108881400A CN108881400A (en) | 2018-11-23 |
| CN108881400B true CN108881400B (en) | 2019-05-28 |
Family
ID=64335122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810521643.9A Active CN108881400B (en) | 2018-05-28 | 2018-05-28 | A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108881400B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115499346B (en) * | 2022-09-15 | 2024-04-26 | 巨擎网络科技(济南)有限公司 | Speed measuring method, device, equipment and medium for multiple network cards |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104219247A (en) * | 2014-09-22 | 2014-12-17 | 中国科学院上海微系统与信息技术研究所 | Method for accessing IP network to wireless multimedia sensor network |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102359879B (en) * | 2011-06-21 | 2014-05-07 | 北京瑞风协同科技股份有限公司 | Measurement and control system for test network and data acquisition control method |
| CN202676907U (en) * | 2012-08-02 | 2013-01-16 | 成都远望科技有限责任公司 | Multichannel weather radar data collecting device |
| CN103873505B (en) * | 2012-12-12 | 2018-05-11 | 中国电信股份有限公司 | Method, system and the device uploaded offline to cloud storage service device |
| CN103118086A (en) * | 2013-01-22 | 2013-05-22 | 温水生 | Small file buffer memory method and small file buffer memory device based on packet forwarding |
| CN106453126A (en) * | 2016-11-29 | 2017-02-22 | 郑州云海信息技术有限公司 | Virtual machine traffic control method and device |
-
2018
- 2018-05-28 CN CN201810521643.9A patent/CN108881400B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104219247A (en) * | 2014-09-22 | 2014-12-17 | 中国科学院上海微系统与信息技术研究所 | Method for accessing IP network to wireless multimedia sensor network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108881400A (en) | 2018-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11061580B2 (en) | Storage device and controllers included in storage device | |
| EP3285187B1 (en) | Optimized merge-sorting of data retrieved from parallel storage units | |
| US20150074672A1 (en) | Asynchronous scheduling informed by job characteristics and anticipatory provisioning of data for real-time, parallel processing | |
| CN111427844B (en) | Data migration system and method for file hierarchical storage | |
| CN104391737B (en) | The optimization method of load balance in cloud platform | |
| CN110457281A (en) | Data processing method, device, equipment and medium | |
| US10108672B2 (en) | Stream-based object storage solution for real-time applications | |
| US20220179585A1 (en) | Management of Idle Time Compute Tasks in Storage Systems | |
| CN109240997A (en) | A kind of upload store method, system and the client of file | |
| CN107273200B (en) | Task scheduling method for heterogeneous storage | |
| CN104717247A (en) | Method and system for dynamically scheduling storage resources in cloud storage system | |
| CN107888666A (en) | A kind of cross-region data-storage system and method for data synchronization and device | |
| CN111552628A (en) | Distributed pressure measurement system and method for graph database and graph service interface | |
| CN103067479A (en) | Network disk synchronized method and system based on file coldness and hotness | |
| US20200034040A1 (en) | Data Architecture Based on Sub-allocation and References from Fragmented Data Blocks | |
| CN108881400B (en) | A kind of system for realizing that multi-channel data uploads in real time based on more network interface cards | |
| CN104410666A (en) | Method and system for implementing heterogeneous storage resource management under cloud computing | |
| CN105049524B (en) | A method of the large-scale dataset based on HDFS loads | |
| CN105005517B (en) | The storage and backup system and method for call center's recording file | |
| KR20180047828A (en) | Data processing system and data processing method | |
| CN106412513B (en) | Processing system for video and processing method | |
| CN102523410A (en) | Method for writing video data and video data storage equipment | |
| CN106610967A (en) | Method and apparatus for reading and writing video data in NAS device | |
| US20140344328A1 (en) | Data collection and distribution management | |
| CN103902745A (en) | Data acquisition method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |