CN106156079A - Daily record data treating method and apparatus - Google Patents
Daily record data treating method and apparatus Download PDFInfo
- Publication number
- CN106156079A CN106156079A CN201510149733.6A CN201510149733A CN106156079A CN 106156079 A CN106156079 A CN 106156079A CN 201510149733 A CN201510149733 A CN 201510149733A CN 106156079 A CN106156079 A CN 106156079A
- Authority
- CN
- China
- Prior art keywords
- daily record
- record data
- source
- result
- memory queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a kind of daily record data treating method and apparatus.The method includes: real-time collecting is from the daily record data in multiple sources；According to the source belonging to the daily record data of real-time collecting, the daily record data of collection is divided to the memory queue corresponding with source；Cutting into slices by the daily record data in the memory queue corresponding with source is cut in real time daily record data, the daily record data section to being cut into uses batch processing, collects the result of batch processing and obtains daily record data result.The present invention improves the real-time that daily record data processes.
Description
Technical field
The present invention relates to the information processing technology, particularly relate to a kind of daily record data treating method and apparatus.
Background technology
In areas of information technology, utilize daily record to monitor information equipment operation conditions, be the most commonplace
Means.
Existing daily record data handling process is usually, the daily record produced when gather information equipment runs, and turns
Change predetermined form into be stored in data base.At set intervals, by the report being mounted with Processing Algorithm
The daily record of form predetermined in engine reading database, obtains daily record data result.This daily record number
Daily record data diagnosis, daily record data prediction etc. are included according to process.Need every one owing to daily record data processes
The section time is carried out, it is ensured that the real-time that daily record data does not processes.
Summary of the invention
In view of this, one of problem that one embodiment of the present of invention solves is to improve daily record data to process
Real-time.
According to one embodiment of present invention, it is provided that a kind of daily record data processing method, including: in real time
Collect the daily record data from multiple sources；According to the source belonging to the daily record data of real-time collecting, by collect
Daily record data divides to the memory queue corresponding with source；By in real time by the memory queue corresponding with source
Daily record data is cut into daily record data section, and the daily record data section to being cut into uses batch processing, to batch processing
Result carry out collecting to obtain daily record data result.
Alternatively, this daily record data processing method includes: notify described daily record data result.
Alternatively, notify that the step of described daily record data result includes: processed by described daily record data
Result is stored on cloud storage and accesses for user.
Alternatively, notify that the step of described daily record data result includes: processed by described daily record data
Result is sent to the notice center of the user of iOS operating system, this notice center notice operate to iOS
The user of system.
Alternatively, this daily record data processing method also includes: judge the day in the memory queue corresponding with source
Will data need to process in real time or need not to process in real time.Described by real time by corresponding with source
Daily record data in memory queue is cut into daily record data section, uses the daily record data section being cut at criticizing
Reason, collects the result of batch processing and obtains the step of daily record data result and be in response to judge
Going out the daily record data in the memory queue corresponding with source is to need the daily record data processed in real time just to perform.
Alternatively, this daily record data processing method also includes: in response to judging the internal memory team corresponding with source
Daily record data in row is the daily record data that need not process in real time, stores corresponding with source in reservoir
Daily record data in memory queue, in order to later to the daily record number in the memory queue corresponding with source of storage
According to carrying out centralized daily record data process.
Alternatively, this daily record data processing method also includes: store the internal memory corresponding with source in reservoir
Daily record data in queue, in order to later the daily record data in the memory queue corresponding with source of storage is entered
The centralized daily record data of row processes.
Alternatively, the daily record data section to being cut into uses batch processing and carries out the result of batch processing
Collect employing machine learning method.
Alternatively, machine learning method includes: cut into slices the daily record data being cut into as pending daily record
Data, are separately sent to the bottom layer node of machine learning network, the wherein bottom layer node of machine learning network
Receive pending daily record data, the top mode output journal data processed result of machine learning network.
Alternatively, described daily record data processing method is daily record data diagnostic method or daily record data prediction side
Method.
According to one embodiment of present invention, it is provided that a kind of daily record data processing means, including: collect
Unit, is configured to the real-time collecting daily record data from multiple sources；Queue forms unit, is configured to
According to the source belonging to the daily record data of real-time collecting, the daily record data of collection is divided to the internal memory corresponding with source
In queue；Daily record data result obtains unit, is configured to the internal memory corresponding with source in real time
Daily record data in queue is cut into daily record data section, and the daily record data section to being cut into uses batch processing,
The result of batch processing is collected and obtains daily record data result.
Alternatively, this daily record data processing means includes: notification unit, is configured to notify described daily record
Data processed result.
Alternatively, notification unit is configured to: described daily record data result is stored in cloud storage
Upper for user's access.
Alternatively, notification unit is configured to: described daily record data result is sent to iOS operation
The notice center of the user of system, by the user of this notice center notice to iOS operating system.
Alternatively, this daily record data processing means also includes: judging unit, is configured to judge and source pair
Daily record data in the memory queue answered needs to process in real time or need not to process in real time；And it is described
Daily record data result obtains unit and is configured to respond to judge in the memory queue corresponding with source
Daily record data be the daily record datas needing to process in real time, by real time by the memory queue corresponding with source
Daily record data be cut into daily record data section, to be cut into daily record data section use batch processing, to batch at
The result of reason carries out collecting to obtain daily record data result.
Alternatively, this daily record data processing means also includes: the first batch processing unit, is configured to ring
Ying Yu judges that the daily record data in the memory queue corresponding with source is the daily record number that need not process in real time
According to, reservoir is stored the daily record data in the memory queue corresponding with source, in order to later to storage
The daily record data in memory queue corresponding with source carries out centralized daily record data process.
Alternatively, this daily record data processing means also includes: the second batch processing unit, is configured to
Reservoir is stored the daily record data in the memory queue corresponding with source, in order to later to store with source pair
Daily record data in the memory queue answered carries out centralized daily record data process.
Alternatively, the daily record data section to being cut into uses batch processing and carries out the result of batch processing
Collect employing machine learning method.
Alternatively, machine learning method includes: cut into slices the daily record data being cut into as pending daily record
Data, are separately sent to the bottom layer node of machine learning network, the wherein bottom layer node of machine learning network
Receive pending daily record data, the top mode output journal data processed result of machine learning network.
Alternatively, described daily record data processing method is the daily record data diagnostic equipment or daily record data prediction dress
Put.
It not that the daily record data of real-time collecting is put into post processing in data base due to the embodiment of the present invention,
And be directly that a memory queue is safeguarded, in real time from memory queue in each daily record data source in internal memory
In take daily record data, section, to each daily record data section use batch processing, and to batch processing result converge
Always, so, being achieved that real-time collecting daily record data, another side has obtained daily record data the most in real time
Result, improves the real-time that daily record data processes, it also avoid in data volume the biggest simultaneously
In the case of use database purchase, the mode that afterwards processes to take a large amount of storage, calculate the hardware bottle in space
Neck problem.
Accompanying drawing explanation
Other feature, feature, advantage and the benefit of the present invention passes through the detailed description below in conjunction with accompanying drawing will
Become more fully apparent.
Fig. 1 represents cloud computing environment according to an embodiment of the invention.
Fig. 2 is the flow chart of daily record data processing method according to an embodiment of the invention.
Fig. 3 is the flow chart of daily record data processing method according to another embodiment of the invention.
Fig. 4 is the flow chart of daily record data processing method according to another embodiment of the invention.
Fig. 5 is the flow chart of daily record data processing method according to another embodiment of the invention.
Fig. 6 is be embodied as the one of daily record data processing method according to another embodiment of the invention
The schematic diagram of individual example.
Fig. 7 be daily record data processing method according to another embodiment of the invention be embodied as another
The schematic diagram of one example.
Fig. 8 is the block diagram of daily record data processing means according to an embodiment of the invention.
Fig. 9 is the block diagram of daily record data processing means according to another embodiment of the invention.
Figure 10 is the block diagram of daily record data processing means according to another embodiment of the invention.
Figure 11 is the block diagram of daily record data processing means according to another embodiment of the invention.
Figure 12 is the structure chart of daily record data processing equipment according to an embodiment of the invention.
Detailed description of the invention
It is more fully described the preferred implementation of the disclosure below with reference to accompanying drawings.Although accompanying drawing shows
The preferred implementation of the disclosure, however, it is to be appreciated that may be realized in various forms the disclosure and not
Should be limited by embodiments set forth herein.On the contrary, it is provided that these embodiments are to make the disclosure
More thorough and complete, and complete for the scope of the present disclosure can be conveyed to those skilled in the art
Member.
It is understood in advance that, although the disclosure includes the detailed description about cloud computing, but described in it
The realization of technical scheme is but not limited to cloud computing environment, but can be in conjunction with currently known or later exploitation
Any other type of computing environment and realize.
Cloud computing is a kind of service offering pattern, for the shared configurable calculating resource pool side of carrying out
Just, on-demand network accesses.The configurable resource that calculates is with minimum management cost or can to carry with service
Donor carries out minimum mutual just energy rapid deployment and the resource of release, such as, can be network, Netowrk tape
Width, server, process, internal memory, store, apply, virtual machine and service.
With reference now to Fig. 1, which show exemplary cloud computing environment 50.As it can be seen, cloud computing
Environment 50 include the local computing device that cloud computing consumer uses can communicate therewith one of letter or
Multiple cloud nodes 10, local computing device can be such as personal digital assistant (PDA) or mobile phone
54A, desktop computer 54B, notebook computer 54C and/or Automotive Computer System 54N.Cloud node
Can be in communication with each other between 10.Can include but not limited to privately owned cloud as above, community Cloud,
Cloud node 10 is carried out thing by one or more network of public cloud or mixed cloud or combinations thereof
Reason or virtual group (not shown).So, the consumer of cloud is without tieing up on local computing device
Protect that the architecture that resource just can request that cloud computing environment 50 provides i.e. services (IaaS), platform i.e. services
And/or software i.e. services (SaaS) (PaaS).Should be appreciated that all kinds of calculating equipment 54A-N that Fig. 1 shows
Be only schematically, cloud node 10 and cloud computing environment 50 can with any type of network on and/
Or any type of calculating equipment (such as the using web browser) communication that network addressable connects.
Fig. 2 is the flow chart of daily record data processing method 1 according to an embodiment of the invention.
Daily record is information equipment or the information of industrial equipment the most self registering expression running status
Or the data produced when running, it includes error messages, service data, use data etc..Information equipment
It is for exchanging, process the equipment of information in information network, such as the computer node in communication network.Work
Industry equipment is the equipment used in industry, such as lathe, motor etc..Error messages is information equipment or industry
Automatically the type of error that carries out when equipment is made mistakes, error reason, wrong time etc..Such as user calculates
Suddenly cannot access webpage during machine online, its type of error is to access webpage, and error reason is current
Wifi signal is more weak, wrong time is the time started that cannot access webpage.Service data is information equipment
Or the self registering data relevant with operation, such as Digit Control Machine Tool are operationally wanted during industrial equipment operation
Measure some parameters of the part to be processed on lathe thus determine some operational factors when lathe runs,
The parameter of the part to be processed now measured and the operational factor of the parameter decision according to part to be processed, all
The service data produced when being to run.Using data is instruction information equipment or industrial equipment behaviour in service
Data, such as monitor makes employment, use time, place etc..Daily record produces in units of bar,
Such as lathe is made mistakes, with regard to a newly generated daily record including type of error, error reason etc..
Daily record data is the general name to the daily record produced, and it is not in units of bar.
Daily record data processes and includes daily record data diagnosis or daily record data prediction etc..Daily record data diagnosis refers to
Whether information equipment or industrial equipment according to daily record data analysis generation daily record there occurs fault.Daily record number
It is predicted and refer to predict that the information equipment of generation daily record or industry set according to former and current daily record data
The standby daily record that later can produce, thus predict the information equipment of generation daily record or the operation that industrial equipment is later
Situation, in order to for its reasonable arrangement resource etc..
In step 410, real-time collecting is from the daily record data in multiple sources.
Source refers to the source of daily record data.It can be an information equipment or an industrial equipment, it is possible to
To be the set of multiple information equipment or industrial equipment, such as, lathe and the motor powered for lathe are closed
It is used as a source.Using an information equipment or an industrial equipment as a source, collect opening of producing
Pin will be bigger.Therefore, it can the set of multiple information equipments or industrial equipment as a source,
Multiple information equipments or the daily record of industrial equipment generation in this source are sent to a unified collection
Entrance, it is possible to from this collection entrance real-time collecting from the daily record data in multiple sources.In this case,
Information equipment or the mark of industrial equipment producing it is had in daily record data.
Such as, Fig. 6 shows four source 801-804, and the most each source is a lathe respectively and is it
The motor of power supply.
In one embodiment, it is possible to use flume and sqoop of such as apache company two exploitation
Source code should be used for the real-time collecting daily record data from multiple sources.
At step 420, according to the source belonging to the daily record data of real-time collecting, the daily record data that will collect
Divide to the memory queue the most corresponding with source.
Such as, in figure 6, owing to there being four source 801-804, it is that four source 801-804 safeguard respectively
Corresponding memory queue 811-814.When real-time collecting is to after a daily record in source 801, incite somebody to action
This daily record divides to the memory queue 811 corresponding with source 801.When real-time collecting is to from source 802
Article one, after daily record, this daily record is divided to the memory queue 812 corresponding with source 802.By that analogy.
According to the source belonging to the daily record data of real-time collecting, the daily record data of collection is divided to corresponding with source
Memory queue such as can be realized by a kind of middleware product being called kafka.
The caching that this internal queues is actually safeguarded respectively for each source.It such as follows first in first out
Method.Take from internal queues daily record carry out subsequent treatment time, first from the daily record entering internal queues at first
Start to take, aim at the day taken away from internal queues internal memory does not exists.Therefore, the present invention implements
Example only uses the mode of internal memory without memorizer or database purchase daily record, greatly reduces daily record data
Storage and process expense.
In step 430, by real time the daily record data in the memory queue corresponding with source being cut into daily record
Data slicer, the daily record data section to being cut into uses batch processing, collects to come to the result of batch processing
Obtain daily record data result.
Daily record data section is the set of some daily record compositions.
According to the sequencing entering memory queue, the daily record data in memory queue can be cut into slices.Example
As, the memory queue corresponding with source 1 has 10 daily records at present, by the priority entering memory queue
Order is respectively daily record 1, daily record 2 ... daily record 10.Process to daily record data section is by Fig. 1
Cloud node 10 in middle cloud computing environment 50 is carried out.Assume cloud computing environment 50 can be used at present
5 are had, the i.e. first to the 5th cloud node in the cloud node 10 that daily record data section is carried out bottom layer treatment.
Daily record 1,2 can be sent to the first cloud node as a daily record data section, daily record 3,4 is made
It is that a daily record data section is sent to the second cloud node, daily record 5,6 is cut as a daily record data
Sheet is sent to the 3rd cloud node, and as a daily record data section, daily record 7,8 is sent to the 4th cloud joint
Point, is sent to the 5th cloud node using daily record 9,10 as a daily record data section.
According to the type of daily record, the daily record data in memory queue can be cut into slices.Such as, right with source 1
The memory queue answered has 10 daily records at present, have 2 error messages daily records, 5 service data daily records,
Article 3, use data logging.Assume cloud computing environment 50 to may be used at present daily record data section is carried out
The cloud node 10 of bottom layer treatment has 4, i.e. first to fourth cloud node.Can be by 2 error messages
Daily record is sent to the first cloud node as a daily record data section, and 5 service data daily records are divided into two
The section of individual daily record data is sent respectively to second, third cloud node, is sent to by 3 use data loggings
4th cloud node.
The most batches of process of batch processing, it is that different daily record data sections is sent to different process joints
Point, the variant node that processes just can process with these different daily record data sections of parallel processing
After the variant result processing node is collected.Owing to it need not wait until that previous daily record data is cut
Sheet has processed and has just started to process next daily record data section, which increases the efficiency that daily record data processes.
Aforementioned 10 daily records in memory queue are respectively classified into the section of 5 or 4 daily record datas it are sent to 5 or 4
The individual cloud nodal parallel for bottom layer treatment processes, and is actually a kind of batch processing.
Collect and refer to, after the cloud node processing daily record data of bottom layer treatment is cut into slices, result be reported
To the cloud node processed for last layer, after the cloud node processing that last layer processes, it is processed to knot again
Fruit is reported to the cloud node processed for more last layer, until being reported to the cloud joint processed for top
Point, is provided result by the cloud node processed for top.
Fig. 6 and 9 outputs two examples that step 430 specifically performs respectively.
In figure 6, source 801 is lathe 1 and the motor 1 for its power supply, and source 802 is lathe 2 and is
Its power supply motor 2, source 803 be lathe 3 and for its power motor 3, source 804 be lathe 4 and
The motor 4 powered for it.Memory queue 811-814 is memory queue corresponding with source 801-804 respectively.
Cloud node 821-835 is each equivalent to the cloud node 10 in Fig. 1 cloud computing environment 50.According to daily record
Daily record data in memory queue is cut into slices by type.Such as, in memory queue 811, by lathe 1 and
The error messages daily record produced for its motor 1 powered is assigned to and is sent to cloud joint in the section of daily record data
Point 821, a daily record number is assigned in the service data daily record that the motor 1 by lathe 1 and for its power supply produces
It is sent to cloud node 822, the use data that the motor 1 by lathe 1 and for its power supply produces according in section
Daily record is assigned in a daily record data section and is sent to cloud node 823.Cloud node 821-823 is according to preset
Algorithm respectively error messages, service data, use data are carried out preliminary analysis, by respective tentatively
Analysis result is all sent to cloud node 826-827.Cloud node 826 is responsible for lathe according to the algorithm of threshold value
1 relevant error messages, service data, use data are comprehensively analyzed.Cloud node 827 is according to threshold
The responsible error messages relevant to motor 1 of the algorithm of value, service data, use data are comprehensively analyzed.
Cloud node 826-827 is respectively by lathe 1 and the error messages of motor 1, service data, use data
Comprehensive analysis results be sent to cloud node 831.Cloud node 831 according to the algorithm of threshold value to cloud node
Lathe 1 and the error messages of motor 1, service data, the comprehensive of use data are analyzed by 826-827
Result is comprehensively analyzed, and obtains the diagnostic result whether entirety to lathe 1 and motor 1 breaks down.
For memory queue 812-814, the process being similar to can be taked.
In the figure 7, the daily record data section to being cut into uses batch processing and enters the result of batch processing
Row collects employing machine learning method.Machine learning method has mature technology at present.
The daily record data being cut into is cut into slices as pending daily record data, is separately sent to machine learning net
Bottom layer node 836-840 of network, wherein bottom layer node 836-840 of machine learning network receives pending
Daily record data, the top mode 847 output journal data processed result of machine learning network.Cloud node
836-847 constitutes a machine learning network.Cloud node 836-847 is each equivalent to Fig. 1 cloud computing ring
Cloud node 10 in border 50.In the figure 7, cloud node 836-840 constitutes the first of machine learning network
Layer L1, they go out feature the most on the one hand from respective input extracting data, and (this feature is probably
Abstract, not there is realistic meaning) as output result, to the second layer L2's of machine learning network
All cloud node 841-844 send.Each of cloud node 841-844 receives from ground floor
After the output result of all cloud node 836-840, to all cloud node 836-840's from ground floor
Output result the most comprehensively extracts feature in a certain respect, and (this feature is probably abstract, does not have reality meaning
Justice) as output result, to all cloud node 845-846 of third layer L3 of machine learning network all
Send.By that analogy, until the cloud node 847 of the 4th layer of L4 of machine learning network to export it comprehensive
The feature extracted, as final daily record data result.This machine learning network is in advance with a large amount of
The sample training of daily record data section becomes so that after inputting the section of any daily record data, it is possible to adaptive
The daily record data result for it should be produced in ground, as daily record data diagnostic result or daily record data are predicted
Result.In the figure 7, source 801 be lathe 1 and for its power supply motor 1, source 802 be lathe 2 and
The motor 2 powered for it, source 803 is lathe 3 and the motor 3 for its power supply, and source 804 is lathe 4
And the motor 4 for its power supply.Memory queue 811-814 is internal memory team corresponding with source 801-804 respectively
Row.At present memory queue 811 such as there are 10 daily records, i.e. daily record 1-10.According to entering internal memory team
Daily record data in memory queue is cut into slices by the sequencing of row.Such as, using daily record 1,2 as one
Daily record data section is sent to cloud node 836, daily record 3,4 is sent to as a daily record data section
Cloud node 837, is sent to cloud node 838, by daily record using daily record 5,6 as a daily record data section
7,8 it is sent to cloud node 839, using daily record 9,10 as a daily record as a daily record data section
Data slicer is sent to cloud node 840.In each daily record data is cut into slices, may there is error messages daily record,
It is likely to service data daily record, it is also possible to have use data logging.The machine that cloud node 836-847 is constituted
The section of these daily record datas is processed by device learning network, obtains final daily record data result,
The diagnostic result such as whether entirety of lathe 1 and motor 1 broken down.For memory queue
812-814, can take the process being similar to.
Fig. 3 shows the flow chart of daily record data processing method 4 in accordance with another embodiment of the present invention.
Compared with Fig. 2, Fig. 3 also includes step 440, i.e. notifies described daily record data result.
Notify that described daily record data result can use directly to this daily record data result of needs
User send out mail, note or the mode being directly displayed on the special equipment of this user.Need this day
The user of will data processed result e.g., is lathe and the situation of the motor for its power supply in above-mentioned source
Under, lathe and the attendant of motor.
Notify the one of described daily record data result preferably, described daily record data is processed
Result is stored on cloud storage 880 and accesses for user.Advantage of this is that, enable numerous user
Share this daily record data result, improve the popularity that this daily record data result utilizes.Such as,
In the case of above-mentioned source is lathe and the motor for its power supply, the dimension of this lathe and motor may be not only
The personnel of protecting need this daily record data result, this lathe possible and all management of motor unit one belongs to
Layer and may use all workmans of this lathe and motor, is required for this daily record data result.
If going notice one by one, notice efficiency is the lowest.It is stored in the mode on cloud storage 880, improves
Notice efficiency.
Notify the another kind of described daily record data result preferably, at described daily record data
Reason result is sent to the notice center 881 of the user of iOS operating system, by this notice center notice to iOS
The user 890 of operating system.At present, the terminal using iOS operating system gets more and more.Notice center
It is the centralization equipment of the notice for application introduced in iOS5.It not be used in the terminal of each user
Any application of upper installation.As long as by the notice center of message informing to the user of iOS operating system, notice
Center will be automatically by each user of this message informing to iOS operating system.Utilize iOS operating system
This feature at the notice center of user, that can improve that daily record data result of the present invention is utilized is wide
General property.
Fig. 4 shows the flow chart of daily record data processing method 4 in accordance with another embodiment of the present invention.
Compared with Fig. 2, Fig. 4 adds step 425, i.e. judges the daily record number in the memory queue corresponding with source
According to type be to need the daily record data processed in real time or need not the daily record data processed in real time.As front
Described, daily record includes error messages daily record, service data daily record, uses data logging.Either make mistakes
Message logging, service data daily record, or all each some needs to process in real time to use data logging,
I.e. must pay close attention at that time, another part need not process in real time, processes the most afterwards and is still able to do in time.
Which daily record data needs to process in real time, and which daily record data need not process in real time, is in advance by user
Regulation also solidifies with the form of program.
The daily record data that step 430 is in response to judge in the memory queue corresponding with source is to need reality
Time the daily record data that processes just perform.
Fig. 4 also add step 431, i.e. in response to the daily record judged in the memory queue corresponding with source
Data are the daily record datas that need not process in real time, store in the memory queue corresponding with source in reservoir
Daily record data, in order to the daily record data in the memory queue corresponding with source of storage is concentrated later
Formula daily record data processes.
In the centralized i.e. prior art of daily record data process, at set intervals, by being mounted with Processing Algorithm
Report engine read the daily record data of storage in reservoir, obtain the mode of daily record data result.
Need to carry out at set intervals owing to daily record data processes, it is ensured that not daily record data process real-time
Property.But owing to daily record data be divide into the daily record data needing to process in real time by the present embodiment and need not reality
Time process daily record data, just adopt in this way only for the daily record data that need not process in real time,
Therefore the advantage fully utilizing two kinds of daily record data processing modes.
It addition, the embodiment of the present invention can also omit step 425 as shown in Figure 5.It is to say, it is right
All of daily record data the most i.e. uses the mode processed in real time of the present invention, uses again as prior art
The mode focused on after storage.In step 432, reservoir is stored the internal memory team corresponding with source
Daily record data in row, in order to later the daily record data in the memory queue corresponding with source of storage is carried out
Centralized daily record data processes.
The advantage of Fig. 5 is, i.e. meets daily record data and processes the requirement of real-time, meets again and look into afterwards
Look for daily record data or carry out, except the real-time daily record data carried out processes, carrying out other afterwards
The needs that daily record data processes.
As shown in Figure 8, a kind of daily record data processing means 10 according to an embodiment of the invention,
Including: collector unit 1010, it is configured to the real-time collecting daily record data from multiple sources；Queue is formed
Unit 1020, is configured to according to the source belonging to the daily record data of real-time collecting, the daily record data that will collect
Divide to the memory queue the most corresponding with source；Daily record data result obtains unit 1030, is configured to lead to
Cross and in real time the daily record data in the memory queue corresponding with source is cut into daily record data section, to the day being cut into
Will data slicer uses batch processing, collects the result of batch processing and obtains daily record data and process knot
Really.
Alternatively, as it is shown in figure 9, this daily record data processing means also includes: notification unit 1040,
It is configured to notify described daily record data result.
Alternatively, notification unit 1040 is configured to: described daily record data result is stored in cloud
Access for user on memorizer.
Alternatively, notification unit 1040 is configured to: described daily record data result is sent to iOS
The notice center of the user of operating system, by the user of this notice center notice to iOS operating system.
Alternatively, as shown in Figure 10, this daily record data processing means also includes: judging unit 1025,
It is configured to judge that the daily record data in the memory queue corresponding with source needs to process in real time or be not required to
To process in real time.Described daily record data result obtains unit 1030 and is configured to respond to judge
Going out the daily record data in the memory queue corresponding with source is the daily record data needing to process in real time, by real time
Daily record data in the memory queue corresponding with source is cut into daily record data section, to the daily record data being cut into
Section uses batch processing, collects the result of batch processing and obtains daily record data result.
Alternatively, as shown in Figure 10, this daily record data processing means also includes: the first batch processing list
Unit 1031, the daily record data being configured to respond to judge in the memory queue corresponding with source is to need not
The daily record data processed in real time, stores the daily record data in the memory queue corresponding with source in reservoir,
After so that, the daily record data in the memory queue corresponding with source of storage is carried out centralized daily record data
Process.
Alternatively, as shown in figure 11, this daily record data processing means also includes: the second batch processing list
Unit 1032, is configured in reservoir store the daily record data in the memory queue corresponding with source, in order to
Daily record data in the memory queue corresponding with source of storage is carried out centralized daily record data process later.
Alternatively, the daily record data section to being cut into uses batch processing and carries out the result of batch processing
Collect employing machine learning method.
Alternatively, machine learning method includes: cut into slices the daily record data being cut into as pending daily record
Data, are separately sent to the bottom layer node of machine learning network, the wherein bottom layer node of machine learning network
Receive pending daily record data, the top mode output journal data processed result of machine learning network.
Alternatively, described daily record data processing means is the daily record data diagnostic equipment or daily record data prediction dress
Put.
Each unit in Fig. 8-11 can use software, hardware (such as integrated circuit, FPGA etc.),
Or the mode of software and hardware combining realizes.
With reference now to Figure 12, it illustrates and process according to a kind of daily record data of one embodiment of the invention
The structure chart of equipment 14.As shown in figure 12, daily record data processing equipment 14 can include memorizer 1401
With processor 1402.Memorizer 1401 can store executable instruction.Processor 1402 can be according to depositing
The executable instruction that reservoir 1401 is stored, it is achieved the operation performed by unit in Fig. 8-11.
Additionally, the embodiment of the present invention also provides for a kind of machine readable media, on it, storage has executable instruction,
When described executable instruction is performed so that machine performs the operation that processor 1402 is realized.
It will be appreciated by those skilled in the art that each embodiment disclosed above, can be without departing from sending out
Various deformation and change is made in the case of bright essence.Therefore, protection scope of the present invention should be by appended
Claims limit.
Claims (20)
1. a daily record data processing method (4), including:
Real-time collecting is from the daily record data (410) in multiple sources；
According to the source belonging to the daily record data of real-time collecting, the daily record data of collection is divided to corresponding with source
In memory queue (420)；
Cut into slices, to cutting by real time the daily record data in the memory queue corresponding with source being cut into daily record data
The daily record data section become uses batch processing, collects the result of batch processing and obtains at daily record data
Reason result (430).
Daily record data processing method the most according to claim 1, including:
Notify described daily record data result (440).
Daily record data processing method the most according to claim 2, wherein notifies that described daily record data processes knot
The step (440) of fruit includes: described daily record data result be stored on cloud storage for user
Access.
Daily record data processing method the most according to claim 2, wherein notifies that described daily record data processes knot
The step (440) of fruit includes: described daily record data result is sent to the use of iOS operating system
The notice center at family, by the user of this notice center notice to iOS operating system.
Daily record data processing method the most according to claim 1, also includes: judge the internal memory corresponding with source
Daily record data in queue is to need to process in real time, still need not (425) processed in real time；And
It is described by real time the daily record data in the memory queue corresponding with source being cut into daily record data section,
Daily record data section to being cut into uses batch processing, collects the result of batch processing and obtains daily record number
The daily record data being in response to judge in the memory queue corresponding with source according to the step of result is to need
The daily record data to process in real time just performs.
Daily record data processing method the most according to claim 5, also includes:
Need not process in real time in response to the daily record data judged in the memory queue corresponding with source
Daily record data, stores the daily record data in the memory queue corresponding with source, in order to later right in reservoir
Daily record data in the memory queue corresponding with source of storage carries out centralized daily record data process (431).
Daily record data processing method the most according to claim 1, also includes:
The daily record data in the memory queue corresponding with source is stored, in order to later to storage in reservoir
The daily record data in memory queue corresponding with source carries out centralized daily record data process (432).
Daily record data processing method the most according to claim 1, wherein the daily record data section to being cut into is adopted
Result with batch processing and to batch processing collects employing machine learning method.
Daily record data processing method the most according to claim 8, wherein machine learning method includes:
The daily record data being cut into is cut into slices as pending daily record data, is separately sent to machine learning net
The bottom layer node of network, wherein the bottom layer node of machine learning network receives pending daily record data, machine
The top mode output journal data processed result of learning network.
Daily record data processing method the most according to claim 1, wherein said daily record data processing method
It is daily record data diagnostic method or daily record data Forecasting Methodology.
11. 1 kinds of daily record data processing meanss (10), including:
Collector unit (1010), is configured to the real-time collecting daily record data from multiple sources；
Queue forms unit (1020), is configured to according to the source belonging to the daily record data of real-time collecting,
The daily record data of collection is divided to the memory queue corresponding with source；
Daily record data result obtains unit (1030), is configured to corresponding with source in real time
Daily record data in memory queue is cut into daily record data section, uses the daily record data section being cut at criticizing
Reason, collects the result of batch processing and obtains daily record data result.
12. daily record data processing meanss according to claim 11, including:
Notification unit (1040), is configured to notify described daily record data result.
13. daily record data processing meanss according to claim 12, wherein notification unit (1040) is joined
It is set to: described daily record data result is stored on cloud storage and accesses for user.
14. daily record data processing meanss according to claim 12, wherein notification unit (1040) is joined
It is set to: described daily record data result is sent to the notice center of the user of iOS operating system, by
This notice center notice is to the user of iOS operating system.
15. daily record data processing meanss according to claim 11, also include: judging unit (1025),
It is configured to judge that the daily record data in the memory queue corresponding with source needs to process in real time or be not required to
To process in real time；And
Described daily record data result obtains unit (1030) and is configured to respond to judge and source pair
Daily record data in the memory queue answered is the daily record datas needing to process in real time, by real time will be with source pair
Daily record data in the memory queue answered is cut into daily record data section, and the daily record data section to being cut into uses
Batch processing, collects the result of batch processing and obtains daily record data result.
16. daily record data processing meanss according to claim 15, also include:
First batch processing unit (1031), is configured to respond to judge the internal memory team corresponding with source
Daily record data in row is the daily record data that need not process in real time, stores corresponding with source in reservoir
Daily record data in memory queue, in order to later to the daily record number in the memory queue corresponding with source of storage
According to carrying out centralized daily record data process.
17. daily record data processing meanss according to claim 11, also include:
Second batch processing unit (1032), is configured in reservoir store the internal memory corresponding with source
Daily record data in queue, in order to later the daily record data in the memory queue corresponding with source of storage is entered
The centralized daily record data of row processes.
18. daily record data processing meanss according to claim 11, wherein to the daily record data section being cut into
Batch processing and the result to batch processing is used to collect employing machine learning method.
19. daily record data processing meanss according to claim 18, wherein machine learning method includes:
The daily record data being cut into is cut into slices as pending daily record data, is separately sent to machine learning net
The bottom layer node of network, wherein the bottom layer node of machine learning network receives pending daily record data, machine
The top mode output journal data processed result of learning network.
20. daily record data processing meanss according to claim 11, wherein said daily record data processing means
It is the daily record data diagnostic equipment or daily record data prediction means.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510149733.6A CN106156079A (en) | 2015-03-31 | 2015-03-31 | Daily record data treating method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510149733.6A CN106156079A (en) | 2015-03-31 | 2015-03-31 | Daily record data treating method and apparatus |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106156079A true CN106156079A (en) | 2016-11-23 |
Family
ID=57338224
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510149733.6A Pending CN106156079A (en) | 2015-03-31 | 2015-03-31 | Daily record data treating method and apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106156079A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018181217A (en) * | 2017-04-20 | 2018-11-15 | ファナック株式会社 | Acceleration/deceleration control apparatus |
| CN109428946A (en) * | 2017-08-31 | 2019-03-05 | Abb瑞士股份有限公司 | Method and system for Data Stream Processing |
| CN110019071A (en) * | 2017-11-15 | 2019-07-16 | 北大方正集团有限公司 | Data processing method and device |
| CN111400335B (en) * | 2018-12-28 | 2023-09-19 | 上海群蚁信息科技有限公司 | Analysis method and system for cloud environment operation data |
| US12093841B2 (en) | 2019-10-11 | 2024-09-17 | Rohde & Schwarz Gmbh & Co. Kg | Method and system for automatic error diagnosis in a test environment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7457872B2 (en) * | 2003-10-15 | 2008-11-25 | Microsoft Corporation | On-line service/application monitoring and reporting system |
| CN202058147U (en) * | 2011-05-23 | 2011-11-30 | 北京六所和瑞科技发展有限公司 | Distribution type real-time database management system |
| CN103136217A (en) * | 2011-11-24 | 2013-06-05 | 阿里巴巴集团控股有限公司 | Distributed data flow processing method and system thereof |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
-
2015
- 2015-03-31 CN CN201510149733.6A patent/CN106156079A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7457872B2 (en) * | 2003-10-15 | 2008-11-25 | Microsoft Corporation | On-line service/application monitoring and reporting system |
| CN202058147U (en) * | 2011-05-23 | 2011-11-30 | 北京六所和瑞科技发展有限公司 | Distribution type real-time database management system |
| CN103136217A (en) * | 2011-11-24 | 2013-06-05 | 阿里巴巴集团控股有限公司 | Distributed data flow processing method and system thereof |
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018181217A (en) * | 2017-04-20 | 2018-11-15 | ファナック株式会社 | Acceleration/deceleration control apparatus |
| US10649441B2 (en) | 2017-04-20 | 2020-05-12 | Fanuc Corporation | Acceleration and deceleration controller |
| CN109428946A (en) * | 2017-08-31 | 2019-03-05 | Abb瑞士股份有限公司 | Method and system for Data Stream Processing |
| EP3451626A1 (en) * | 2017-08-31 | 2019-03-06 | ABB Schweiz AG | Method and system for data stream processing |
| US10798011B2 (en) | 2017-08-31 | 2020-10-06 | Abb Schweiz Ag | Method and system for data stream processing |
| CN110019071A (en) * | 2017-11-15 | 2019-07-16 | 北大方正集团有限公司 | Data processing method and device |
| CN111400335B (en) * | 2018-12-28 | 2023-09-19 | 上海群蚁信息科技有限公司 | Analysis method and system for cloud environment operation data |
| US12093841B2 (en) | 2019-10-11 | 2024-09-17 | Rohde & Schwarz Gmbh & Co. Kg | Method and system for automatic error diagnosis in a test environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113450065B (en) | Production operation management system and method for wind power plant | |
| Lee et al. | A RFID-based recursive process mining system for quality assurance in the garment industry | |
| CN104616205B (en) | A kind of operation states of electric power system monitoring method based on distributed information log analysis | |
| CN109501834A (en) | A kind of point machine failure prediction method and device | |
| CN106992994A (en) | A kind of automatically-monitored method and system of cloud service | |
| CN105574593B (en) | Track state static detection and control system and method based on cloud computing and big data | |
| CN108259194A (en) | Network failure method for early warning and device | |
| CN105069025A (en) | Intelligent aggregation visualization and management and control system for big data | |
| CN107798395A (en) | A kind of power grid accident signal automatic diagnosis method and system | |
| CN106156079A (en) | Daily record data treating method and apparatus | |
| CN113516244B (en) | Intelligent operation and maintenance method and device, electronic equipment and storage medium | |
| CN102063668A (en) | Auditing method and system for information system | |
| CN109739912A (en) | Data analysing method and system | |
| CN105631612A (en) | System and method of evaluating individual performance and capability of public servant based on big data | |
| CN115908046A (en) | Visual power distribution system based on airport terminal building BIM | |
| CN115733762A (en) | Monitoring system with big data analysis capability | |
| CN110347694A (en) | A kind of apparatus monitoring method based on Internet of Things, apparatus and system | |
| CN112817814A (en) | Abnormity monitoring method, system, storage medium and electronic device | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| CN106487597A (en) | A kind of service monitoring system and method based on Zookeeper | |
| CN118014536A (en) | Rail transit full scene intelligent construction control method and system based on resource constraint | |
| CN109995558A (en) | Failure information processing method, device, equipment and storage medium | |
| Gouthaman et al. | Agile software risk management architecture for IoT-fog based systems | |
| CN113869589A (en) | Power transmission line accident prediction method based on knowledge graph and inspection system | |
| CN118183216A (en) | Fault diagnosis method and system for belt conveyor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161123 |
|
| WD01 | Invention patent application deemed withdrawn after publication |