CN108829646A - Data storage and analytic method, device, system and storage medium - Google Patents
Data storage and analytic method, device, system and storage medium Download PDFInfo
- Publication number
- CN108829646A CN108829646A CN201810413278.XA CN201810413278A CN108829646A CN 108829646 A CN108829646 A CN 108829646A CN 201810413278 A CN201810413278 A CN 201810413278A CN 108829646 A CN108829646 A CN 108829646A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- specifying
- message
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/14—Tree-structured documents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/123—Storage facilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/14—Tree-structured documents
- G06F40/146—Coding or compression of tree-structured data
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/08—Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C5/0841—Registering performance data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses data storage and analytic method, device, system and storage medium, wherein method include：Obtain the message data of tentation data format；Generate data file, record has the message data got in the data file, also, the predetermined position of data file stores the corresponding data specifying-information of message data recorded, to be parsed according to the data specifying-information to the message data in data file.Using scheme of the present invention, data-handling efficiency etc. can be improved.
Description
【Technical field】
The present invention relates to Computer Applied Technology, in particular to data storage and analytic method, device, system and storage is situated between
Matter.
【Background technique】
Automatic driving vehicle refers to through vehicle-mounted sensor-based system and perceives vehicle-periphery, and is obtained according to perception
Road, vehicle location and obstacle information etc., the steering and speed of vehicle are controlled, to enable the vehicle to reliably and securely
It is travelled on road.
Data play very crucial and irreplaceable role in automatic driving vehicle.
Robot operating system (ROS, Robot Operating System) is automatic Pilot field widely used one
A Development Framework.In ROS system, the message count generated in automatic driving vehicle operational process is recorded using rosbag file
According to, and the definition of message data is described by msg file, i.e., data specifying-information is recorded by msg file.When needs pair
When message data carries out the operation such as parsing, need just be operated using corresponding msg file.
Fig. 1 is existing message data process of analysis schematic diagram.As shown in Figure 1, specifically including following implementations.
In 101, rosbag file is opened.
In 102, a piece of news data are read.
In 103, it is determined whether there are the corresponding msg files of message data, if so, executing 104, otherwise, execute
105。
In 104, message data is parsed using msg file, executes 102 later.
In 105, the exception that message data of dishing out can not parse executes 102 later.
As can be seen that in the prior art, if msg file cannot be got, cannot achieve the parsing of message data, thus
The normal processing for affecting message data, reduces data-handling efficiency etc..
【Summary of the invention】
In view of this, the present invention provides data storage and analytic method, device, system and storage mediums.
Specific technical solution is as follows：
A kind of date storage method, including：
Obtain the message data of tentation data format；
Data file is generated, recording in the data file has the message data got, also, the data file
Predetermined position stores the corresponding data specifying-information of message data that is recorded, so as to according to the data specifying-information to institute
The message data stated in data file is parsed.
According to one preferred embodiment of the present invention, the tentation data format includes：Protocol cache protobuf data format.
According to one preferred embodiment of the present invention, the predetermined position includes：Head.
According to one preferred embodiment of the present invention, the data specifying-information includes：Sequence turns to the data description of character string
Information.
According to one preferred embodiment of the present invention, the different messages data of same type correspond to same data specifying-information.
According to one preferred embodiment of the present invention, the data file includes：Record record file.
A kind of data analysis method, including：
When needing to parse any message data recorded in data file, from the pre-determined bit of the data file
It sets and obtains the corresponding data specifying-information of the message data；The predetermined position of the data file stores the message recorded
The corresponding data specifying-information of data；
The message data is parsed according to the data specifying-information.
According to one preferred embodiment of the present invention, described to obtain the message data pair from the predetermined position of the data file
Before the data specifying-information answered, further comprise：
Determine whether to have obtained the data specifying-information；
If so, being parsed according to the data specifying-information obtained to the message data；
If it is not, then obtaining the data specifying-information from the predetermined position of the data file, and retouched according to the data
Information is stated to parse the message data.
According to one preferred embodiment of the present invention, described to determine whether that having obtained the data specifying-information includes：
Determine whether the user of the message data specifies the data specifying-information, if so, determination has obtained
The data specifying-information, if not, it is determined that the data specifying-information has not been obtained.
A kind of data storage device, including：First acquisition unit and storage unit；
The first acquisition unit, for obtaining the message data of tentation data format；
The storage unit, for generating data file, record has the message data got in the data file, and
And the predetermined position of the data file stores the corresponding data specifying-information of message data recorded, so as to according to institute
Data specifying-information is stated to parse the message data in the data file.
According to one preferred embodiment of the present invention, the tentation data format includes：Protocol cache protobuf data format.
According to one preferred embodiment of the present invention, the predetermined position includes：Head.
According to one preferred embodiment of the present invention, the data specifying-information includes：Sequence turns to the data description of character string
Information.
According to one preferred embodiment of the present invention, the different messages data of same type correspond to same data specifying-information.
According to one preferred embodiment of the present invention, the data file includes：Record record file.
A kind of data analysis device, including：Second acquisition unit and resolution unit；
The second acquisition unit, for when needing to parse any message data recorded in data file,
The corresponding data specifying-information of the message data is obtained from the predetermined position of the data file；The data file makes a reservation for
Position stores the corresponding data specifying-information of message data recorded；
The resolution unit, for being parsed according to the data specifying-information to the message data.
According to one preferred embodiment of the present invention, the second acquisition unit is further used for,
Before obtaining the corresponding data specifying-information of the message data from the predetermined position of the data file, determine
Whether the data specifying-information has been obtained；
If so, the resolution unit is notified to be carried out according to the data specifying-information obtained to the message data
Parsing；
If it is not, then obtaining the data specifying-information from the predetermined position of the data file, and notify that the parsing is single
Member parses the message data according to the data specifying-information.
According to one preferred embodiment of the present invention, the second acquisition unit determines whether the user of the message data refers to
The data specifying-information is determined, if so, determination has obtained the data specifying-information, if not, it is determined that have not been obtained described
Data specifying-information.
A kind of automatic driving vehicle data processing system, including：
Data storage device as described above, and, data analysis device as described above.
A kind of computer equipment, including memory, processor and be stored on the memory and can be in the processor
The computer program of upper operation, the processor realize method as described above when executing described program.
A kind of computer readable storage medium is stored thereon with computer program, real when described program is executed by processor
Now method as described above.
It can be seen that based on above-mentioned introduction using scheme of the present invention, it can be according to the tentation data format got
Message data generates data file, and record has the message data got, also, the predetermined position of data file in data file
The corresponding data specifying-information of message data recorded is stored, in this way, subsequent can be according to the data specifying-information stored
To be parsed to the message data in data file, that is to say, that in scheme of the present invention, message data and data description
Information can store in the data file simultaneously, the problem of so as to avoid data specifying-information can not be got in the prior art,
And then ensured the normal processing of message data, improve data-handling efficiency etc..
【Detailed description of the invention】
Fig. 1 is existing message data process of analysis schematic diagram.
Fig. 2 is the flow chart of date storage method embodiment of the present invention.
Fig. 3 is the flow chart of data analysis method embodiment of the present invention.
Fig. 4 is the flow chart of data analysis method preferred embodiment of the present invention.
Fig. 5 is data recording schematic diagram of the present invention.
Fig. 6 is data playback of the present invention and analysis mode schematic diagram.
Fig. 7 is the composed structure schematic diagram of data storage device embodiment of the present invention.
Fig. 8 is the composed structure schematic diagram of data analysis device embodiment of the present invention.
Fig. 9 shows the block diagram for being suitable for the exemplary computer system/server 12 for being used to realize embodiment of the present invention.
【Specific embodiment】
Aiming at the problems existing in the prior art, a kind of storage of data and analysis mode are proposed in the present invention.
In order to be clearer and more clear technical solution of the present invention, hereinafter, referring to the drawings and the embodiments, to institute of the present invention
The scheme of stating is further described.
Obviously, described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on the present invention
In embodiment, those skilled in the art's all other embodiment obtained without creative efforts, all
Belong to the scope of protection of the invention.
Fig. 2 is the flow chart of date storage method embodiment of the present invention.As shown in Fig. 2, including realizing in detail below
Mode.
In 201, the message data of tentation data format is obtained.
In 202, data file is generated, wherein record has the message data got, also, the pre-determined bit of data file
Set the corresponding data specifying-information of message data that is recorded of storage, so as to according to the data specifying-information to data file
In message data parsed.
Preferably, tentation data format can be protocol cache (protobuf) data format.
For example, the message data that generated in automatic driving vehicle operational process can be carried out with the data format of protobuf
It saves.
Protobuf is the abbreviation of protocol buffer, is a kind of structural data storage format of portable and effective, with
Platform is unrelated, language is unrelated, expansible, can be applied to the numerous areas such as communications protocol and data storage.
Protobuf has forward and backward compatibility, can support to be fitted automatically between the message data of different editions
Match.
Data file can be generated according to each message data got, record has each message data in data file, in addition,
The predetermined position of data file stores the corresponding data specifying-information of message data recorded.
Preferably, data file can be to record (record) file, i.e. the suffix of data file is record, is got
The message data of each protobuf data format is recorded in record file.
Since the corresponding data specifying-information of the message data of same type is identical, it can be without being directed to every message
Data, store its corresponding data specifying-information respectively, but can the different messages data of same type correspond to same data and retouch
Information is stated, so that the data volume stored needed for reducing, avoids data redundancy etc..
It illustrates：100,000 message datas are had recorded in data file altogether, this 100,000 message datas are belonging respectively to 5 kinds
Different type stores its corresponding data specifying-information then can then be directed to this 5 seed type respectively.
How the data specifying-information of different types of message data is obtained with no restriction.May include in data specifying-information
Constituted mode includes which field, field location, field type etc..
The data specifying-information sequence that can be will acquire turns to character string, and then the data for storing character string forms describe letter
Breath.
In addition, data specifying-information can be stored in the head of data file.Other than including data specifying-information, data
The head of file may also include the format of file, structural information etc..
It is the introduction about date storage method of the present invention above, based on above-mentioned storage method, it can be achieved that the parsing of data.
Fig. 3 is the flow chart of data analysis method embodiment of the present invention.As shown in figure 3, including realizing in detail below
Mode.
In 301, when needing to parse any message data recorded in data file, from the pre- of data file
Determine the corresponding data specifying-information of position acquisition message data；The predetermined position of data file stores the message data recorded
Corresponding data specifying-information.
In 302, message data is parsed according to data specifying-information.
It, can be from data file when needing to parse any message data in data file in the present embodiment
Predetermined position such as head obtains the corresponding data specifying-information of message data, the i.e. corresponding data description letter of the type of message data
Breath can complete the parsing to message data later according to the data specifying-information got.
In some cases, before obtaining data specifying-information in the manner described above, data description letter may have been got
Breath, if the user of message data specifies data specifying-information, then then no longer needing to obtain data description in the manner described above
Information, but directly message data can be parsed according to the data specifying-information obtained.
Based on above-mentioned introduction, Fig. 4 is the flow chart of data analysis method preferred embodiment of the present invention.As shown in figure 4,
Including implementation in detail below.
In 401, data file is opened.
In 402, a piece of news data are read.
In 403, it is determined whether the corresponding data specifying-information of message data has been obtained, if so, 404 are executed, otherwise,
Execute 405.
Such as, it may be determined that whether the user of message data specifies data specifying-information, if so, determination has obtained number
According to description information, 404 are executed later, if not, it is determined that data specifying-information has not been obtained, executes 405 later.
In 404, message data is parsed according to the data specifying-information obtained, executes 402 later.
Message is described according to the data obtained, according to existing way building data object (object) and is parsed in message
Hold etc., complete parsing.
In 405, the corresponding data specifying-information of message data is obtained from data file.
While reading message data in 402, the type information of message data can be obtained, according to the class of message data
Type obtains corresponding data specifying-information from data file.
In 406, message data is parsed according to the data specifying-information got, executes 402 later.
Message is described according to the data got, can dynamically construct data object, and parse message content etc., thus
Parsing is completed, achievees the effect that self-described.
It should be noted that for the various method embodiments described above, for simple description, therefore, it is stated as a series of
Combination of actions, but those skilled in the art should understand that, the present invention is not limited by the sequence of acts described because
According to the present invention, certain steps can use other sequences or carry out simultaneously.Secondly, those skilled in the art should also know
It knows, the embodiments described in the specification are all preferred embodiments, and related actions and modules is not necessarily of the invention
It is necessary.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, there is no the portion being described in detail in some embodiment
Point, it may refer to the associated description of other embodiments.
The present invention provides the supports of complete functional interface, and corresponding data description can be directly generated by data object
Information, and corresponding data object is constructed by data specifying-information.In addition, the present invention also provides higher level to connect
Mouth encapsulation, application programming interfaces (API, the Application such as recording, broadcasting, reading, parsing including data file
Programming Interface)。
Wherein, record and refer to that the message data that will acquire is saved as a data file, broadcasting refer to by network or its
Its mode is sent.
Fig. 5 is data recording schematic diagram of the present invention.As shown in figure 5, when generating data file, it can be by message
The corresponding data specifying-information of data is stored simultaneously, and data file head is arrived in such as storage.
Fig. 6 is data playback of the present invention and analysis mode schematic diagram.As shown in fig. 6, can be according to the message got
The corresponding data specifying-information of data completes the parsing to message data.Message descriptor shown in Fig. 5 and Fig. 6 is institute
State data specifying-information.
In short, message data and data specifying-information can store simultaneously using in scheme described in above-mentioned each method embodiment
In the data file, so as to avoid data specifying-information can not be got in the prior art the problem of, and can be realized data
The decoupling of storage and data description, the data of recording self-described and can parse etc., and then ensure message data just
Often processing, improves data-handling efficiency etc..
In addition, the data format of self-described has very big help, the provider of data for the open use of data
It only needs to provide data to user, specific algorithm or tool without being additionally provided parsing data.
Further more, being based on protobuf data format, automatic adaptation, data Life Cycle can be carried out between the data of different editions
Phase is long, and the data of early version still can be used partially to be trained or off-line test in new function and algorithm, without
Each more new algorithm all acquires new data, to reduce the cost etc. of acquisition and storage.
It is the introduction about embodiment of the present invention method above, below by way of Installation practice, to scheme of the present invention
It is further described.
Fig. 7 is the composed structure schematic diagram of data storage device embodiment of the present invention.As shown in fig. 7, comprises：First
Acquiring unit 701 and storage unit 702.
First acquisition unit 701, for obtaining the message data of tentation data format.
Storage unit 702, for generating data file, wherein record has the message data got, also, data file
Predetermined position store the corresponding data specifying-information of message data that is recorded, so as to according to the data specifying-information pair
Message data in data file is parsed.
Preferably, tentation data format can be protobuf data format.
For example, the message data that generated in automatic driving vehicle operational process can be carried out with the data format of protobuf
It saves.
Protobuf is the abbreviation of protocol buffer, is a kind of structural data storage format of portable and effective, with
Platform is unrelated, language is unrelated, expansible, can be applied to the numerous areas such as communications protocol and data storage.
Protobuf has forward and backward compatibility, can support to be fitted automatically between the message data of different editions
Match.
Storage unit 702 can generate data file according to each message data got, and there is record in data file respectively disappears
Data are ceased, in addition, the predetermined position of data file stores the corresponding data specifying-information of message data recorded.
Preferably, data file can be record file.In addition, data specifying-information can be stored in the head of data file
Portion.
Since the corresponding data specifying-information of the message data of same type is identical, it can be without being directed to every message
Data, store its corresponding data specifying-information respectively, but can the different messages data of same type correspond to same data and retouch
Information is stated, so that the data volume stored needed for reducing, avoids data redundancy etc..
In addition, the data specifying-information sequence that can be will acquire turns to character string, and then store the data of character string forms
Description information.
Fig. 8 is the composed structure schematic diagram of data analysis device embodiment of the present invention.As shown in figure 8, including：Second
Acquiring unit 801 and resolution unit 802.
Second acquisition unit 801, for when needing to parse any message data recorded in data file, from
The predetermined position of data file obtains the corresponding data specifying-information of message data；The predetermined position storage of data file is remembered
The corresponding data specifying-information of the message data of record.
Resolution unit 802, for being parsed according to data specifying-information to message data.
In some cases, before obtaining data specifying-information in the manner described above, data description letter may have been got
Breath, if the user of message data specifies data specifying-information, then then no longer needing to obtain data description in the manner described above
Information, but directly message data can be parsed according to the data specifying-information obtained.
Correspondingly, second acquisition unit 801 can be further used for, and obtain message data from the predetermined position of data file
Before corresponding data specifying-information, it is determined whether obtained data specifying-information；If so, notice 802 basis of resolution unit
The data specifying-information obtained parses message data；It is retouched if it is not, then obtaining data from the predetermined position of data file
Information is stated, and resolution unit 802 is notified to be parsed according to data specifying-information to message data.
For example, second acquisition unit 801 can determine whether the user of message data specifies data specifying-information, if
It is, it is determined that obtained data specifying-information, if not, it is determined that data specifying-information has not been obtained.
The present invention discloses a kind of automatic driving vehicle data processing systems, including：Data storage as shown in Figure 7
Device, and, data analysis device as shown in Figure 8.
The specific workflow of above-mentioned apparatus and system embodiment please refers to the related description in preceding method embodiment, no
It repeats again.
Fig. 9 shows the block diagram for being suitable for the exemplary computer system/server 12 for being used to realize embodiment of the present invention.
The computer system/server 12 that Fig. 9 is shown is only an example, should not function and use scope to the embodiment of the present invention
Bring any restrictions.
As shown in figure 9, computer system/server 12 is showed in the form of universal computing device.Computer system/service
The component of device 12 can include but is not limited to：One or more processor (processing unit) 16, memory 28 connect not homology
The bus 18 of system component (including memory 28 and processor 16).
Bus 18 indicates one of a few class bus structures or a variety of, including memory bus or Memory Controller,
Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.It lifts
For example, these architectures include but is not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)
Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Computer system/server 12 typically comprises a variety of computer system readable media.These media, which can be, appoints
What usable medium that can be accessed by computer system/server 12, including volatile and non-volatile media, it is moveable and
Immovable medium.
Memory 28 may include the computer system readable media of form of volatile memory, such as random access memory
Device (RAM) 30 and/or cache memory 32.Computer system/server 12 may further include it is other it is removable/no
Movably, volatile/non-volatile computer system storage medium.Only as an example, storage system 34 can be used for reading and writing
Immovable, non-volatile magnetic media (Fig. 9 do not show, commonly referred to as " hard disk drive ").It, can although being not shown in Fig. 9
To provide the disc driver for reading and writing to removable non-volatile magnetic disk (such as " floppy disk "), and it is non-volatile to moving
Property CD (such as CD-ROM, DVD-ROM or other optical mediums) read and write CD drive.In these cases, each drive
Dynamic device can be connected by one or more data media interfaces with bus 18.Memory 28 may include at least one program
Product, the program product have one group of (for example, at least one) program module, these program modules are configured to perform the present invention
The function of each embodiment.
Program/utility 40 with one group of (at least one) program module 42 can store in such as memory 28
In, such program module 42 includes --- but being not limited to --- operating system, one or more application program, other programs
It may include the realization of network environment in module and program data, each of these examples or certain combination.Program mould
Block 42 usually executes function and/or method in embodiment described in the invention.
Computer system/server 12 can also be (such as keyboard, sensing equipment, aobvious with one or more external equipments 14
Show device 24 etc.) communication, it is logical that the equipment interacted with the computer system/server 12 can be also enabled a user to one or more
Letter, and/or with the computer system/server 12 any is set with what one or more of the other calculating equipment was communicated
Standby (such as network interface card, modem etc.) communicates.This communication can be carried out by input/output (I/O) interface 22.And
And computer system/server 12 can also pass through network adapter 20 and one or more network (such as local area network
(LAN), wide area network (WAN) and/or public network, such as internet) communication.As shown in figure 9, network adapter 20 passes through bus
18 communicate with other modules of computer system/server 12.It should be understood that although not shown in the drawings, computer can be combined
Systems/servers 12 use other hardware and/or software module, including but not limited to：Microcode, device driver, at redundancy
Manage unit, external disk drive array, RAID system, tape drive and data backup storage system etc..
The program that processor 16 is stored in memory 28 by operation, at various function application and data
Reason, such as realize the method in Fig. 2 or 3 illustrated embodiments.
The present invention discloses a kind of computer readable storage mediums, are stored thereon with computer program, the program quilt
The method in embodiment as shown in figures 2 and 3 will be realized when processor executes.
It can be using any combination of one or more computer-readable media.Computer-readable medium can be calculating
Machine readable signal medium or computer readable storage medium.Computer readable storage medium for example can be --- but it is unlimited
In system, device or the device of --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, or any above combination.It calculates
The more specific example (non exhaustive list) of machine readable storage medium storing program for executing includes：Electrical connection with one or more conducting wires, just
Taking formula computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In this document, computer readable storage medium can be it is any include or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.
Computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including --- but
It is not limited to --- electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be
Any computer-readable medium other than computer readable storage medium, which can send, propagate or
Transmission is for by the use of instruction execution system, device or device or program in connection.
The program code for including on computer-readable medium can transmit with any suitable medium, including --- but it is unlimited
In --- wireless, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereof
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
Further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
It fully executes, partly execute on the user computer on the user computer, being executed as an independent software package, portion
Divide and partially executes or executed on a remote computer or server completely on the remote computer on the user computer.?
Be related in the situation of remote computer, remote computer can pass through the network of any kind --- including local area network (LAN) or
Wide area network (WAN)-be connected to subscriber computer, or, it may be connected to outer computer (such as mentioned using Internet service
It is connected for quotient by internet).
In several embodiments provided by the present invention, it should be understood that disclosed device and method etc. can pass through
Other modes are realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the unit,
Only a kind of logical function partition, there may be another division manner in actual implementation.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that equipment (can be personal computer, server or the network equipment etc.) or processor (processor) execute the present invention
The part steps of embodiment the method.And storage medium above-mentioned includes：USB flash disk, mobile hard disk, read-only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is various
It can store the medium of program code.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (21)
1. a kind of date storage method, which is characterized in that including：
Obtain the message data of tentation data format；
Data file is generated, record has the message data got in the data file, also, the data file is predetermined
Position stores the corresponding data specifying-information of message data that is recorded, so as to according to the data specifying-information to the number
It is parsed according to the message data in file.
2. the method according to claim 1, wherein
The tentation data format includes：Protocol cache protobuf data format.
3. the method according to claim 1, wherein
The predetermined position includes：Head.
4. the method according to claim 1, wherein
The data specifying-information includes：Sequence turns to the data specifying-information of character string.
5. the method according to claim 1, wherein
The different messages data of same type correspond to same data specifying-information.
6. the method according to claim 1, wherein
The data file includes：Record record file.
7. a kind of data analysis method, which is characterized in that including：
When needing to parse any message data recorded in data file, obtained from the predetermined position of the data file
Take the corresponding data specifying-information of the message data；The predetermined position of the data file stores the message data recorded
Corresponding data specifying-information；
The message data is parsed according to the data specifying-information.
8. the method according to the description of claim 7 is characterized in that
Before the corresponding data specifying-information from the predetermined position of the data file acquisition message data, further
Including：
Determine whether to have obtained the data specifying-information；
If so, being parsed according to the data specifying-information obtained to the message data；
If it is not, then obtaining the data specifying-information from the predetermined position of the data file, and describe to believe according to the data
Breath parses the message data.
9. according to the method described in claim 8, it is characterized in that,
It is described to determine whether that having obtained the data specifying-information includes：
Determine whether the user of the message data specifies the data specifying-information, if so, determine obtained it is described
Data specifying-information, if not, it is determined that the data specifying-information has not been obtained.
10. a kind of data storage device, which is characterized in that including：First acquisition unit and storage unit；
The first acquisition unit, for obtaining the message data of tentation data format；
The storage unit, for generating data file, record has the message data got in the data file, also,
The predetermined position of the data file stores the corresponding data specifying-information of message data recorded, so as to according to the number
The message data in the data file is parsed according to description information.
11. data storage device according to claim 10, which is characterized in that
The tentation data format includes：Protocol cache protobuf data format.
12. data storage device according to claim 10, which is characterized in that
The predetermined position includes：Head.
13. data storage device according to claim 10, which is characterized in that
The data specifying-information includes：Sequence turns to the data specifying-information of character string.
14. data storage device according to claim 10, which is characterized in that
The different messages data of same type correspond to same data specifying-information.
15. data storage device according to claim 10, which is characterized in that
The data file includes：Record record file.
16. a kind of data analysis device, which is characterized in that including：Second acquisition unit and resolution unit；
The second acquisition unit, for when needing to parse any message data recorded in data file, from institute
The predetermined position for stating data file obtains the corresponding data specifying-information of the message data；The predetermined position of the data file
Store the corresponding data specifying-information of message data recorded；
The resolution unit, for being parsed according to the data specifying-information to the message data.
17. data analysis device according to claim 16, which is characterized in that
The second acquisition unit is further used for,
Before obtaining the corresponding data specifying-information of the message data from the predetermined position of the data file, it is determined whether
The data specifying-information is obtained；
If so, the resolution unit is notified to be solved according to the data specifying-information obtained to the message data
Analysis；
If it is not, then obtaining the data specifying-information from the predetermined position of the data file, and notify the resolution unit root
The message data is parsed according to the data specifying-information.
18. data analysis device according to claim 17, which is characterized in that
The second acquisition unit determines whether the user of the message data specifies the data specifying-information, if so,
Then determination has obtained the data specifying-information, if not, it is determined that the data specifying-information has not been obtained.
19. a kind of automatic driving vehicle data processing system, which is characterized in that including：
Data storage device described in any one of claim 10~15, and, described in any one of claim 16~18
Data analysis device.
20. a kind of computer equipment, including memory, processor and it is stored on the memory and can be on the processor
The computer program of operation, which is characterized in that the processor is realized when executing described program as any in claim 1~9
Method described in.
21. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that described program is processed
Such as method according to any one of claims 1 to 9 is realized when device executes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810413278.XA CN108829646A (en) | 2018-05-03 | 2018-05-03 | Data storage and analytic method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810413278.XA CN108829646A (en) | 2018-05-03 | 2018-05-03 | Data storage and analytic method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108829646A true CN108829646A (en) | 2018-11-16 |
Family
ID=64148039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810413278.XA Pending CN108829646A (en) | 2018-05-03 | 2018-05-03 | Data storage and analytic method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108829646A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857742A (en) * | 2019-01-18 | 2019-06-07 | 烽火通信科技股份有限公司 | A kind of big data virtual tables fast display method and system |
| CN110795393A (en) * | 2019-10-31 | 2020-02-14 | 中孚安全技术有限公司 | Method, system, equipment and readable storage medium for analyzing binary format of document |
| CN112991573A (en) * | 2021-02-19 | 2021-06-18 | 北京嘀嘀无限科技发展有限公司 | Automobile data recorder data opening method and related device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077335A (en) * | 2013-05-07 | 2014-10-01 | 腾讯科技（深圳）有限公司 | Methods, devices and system for serializing and deserializing structured data |
| CN104199879A (en) * | 2014-08-21 | 2014-12-10 | 广州华多网络科技有限公司 | Data processing method and device |
| CN105760534A (en) * | 2016-03-10 | 2016-07-13 | 上海晶赞科技发展有限公司 | User-defined serializable data structure, hadoop cluster, server and application method thereof |
| CN106873906A (en) * | 2017-01-04 | 2017-06-20 | 北京百度网讯科技有限公司 | Method and apparatus for managing metamessage |
-
2018
- 2018-05-03 CN CN201810413278.XA patent/CN108829646A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104077335A (en) * | 2013-05-07 | 2014-10-01 | 腾讯科技（深圳）有限公司 | Methods, devices and system for serializing and deserializing structured data |
| CN104199879A (en) * | 2014-08-21 | 2014-12-10 | 广州华多网络科技有限公司 | Data processing method and device |
| CN105760534A (en) * | 2016-03-10 | 2016-07-13 | 上海晶赞科技发展有限公司 | User-defined serializable data structure, hadoop cluster, server and application method thereof |
| CN106873906A (en) * | 2017-01-04 | 2017-06-20 | 北京百度网讯科技有限公司 | Method and apparatus for managing metamessage |
Non-Patent Citations (1)
| Title |
|---|
| RESENTMENT: "《博客园》", 1 April 2017 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109857742A (en) * | 2019-01-18 | 2019-06-07 | 烽火通信科技股份有限公司 | A kind of big data virtual tables fast display method and system |
| CN109857742B (en) * | 2019-01-18 | 2021-03-02 | 烽火通信科技股份有限公司 | Big data virtual table rapid display method and system |
| CN110795393A (en) * | 2019-10-31 | 2020-02-14 | 中孚安全技术有限公司 | Method, system, equipment and readable storage medium for analyzing binary format of document |
| CN112991573A (en) * | 2021-02-19 | 2021-06-18 | 北京嘀嘀无限科技发展有限公司 | Automobile data recorder data opening method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190066671A1 (en) | Far-field speech awaking method, device and terminal device | |
| CN109213611B (en) | Cross-process communication method, device, terminal and storage medium | |
| CN110032459A (en) | Message queue configuration method, device, computer equipment and storage medium | |
| US20050240685A1 (en) | Virtual USB port system and method | |
| WO2020177734A1 (en) | App triggering method, computer device, and storage medium | |
| CN108829646A (en) | Data storage and analytic method, device, system and storage medium | |
| KR101528367B1 (en) | Sound control system and method as the same | |
| CN114385091B (en) | Method and device for realizing network disk drive character, network disk and storage medium | |
| CN109710333A (en) | Processing method, equipment and the computer readable storage medium of boarding application | |
| CN110072268A (en) | Method, equipment and the computer storage medium that wifi hotspot is shared | |
| CN109783345A (en) | A kind of small routine performance test methods and system | |
| CN109976922A (en) | Discovery method, equipment and computer storage medium between small routine platform | |
| CN108551481A (en) | A kind of file uploading method, device, server and storage medium | |
| CN109495549A (en) | Method, equipment and the computer storage medium of work are drawn in a kind of application | |
| CN112055251A (en) | Media data playing method, device, equipment and storage medium | |
| CN109471674A (en) | It spreads its tail information displaying method, computer equipment and storage medium using starting | |
| CN114465956B (en) | Method and device for limiting flow rate of virtual machine, electronic equipment and storage medium | |
| CN109284108A (en) | Date storage method, device, electronic equipment and storage medium | |
| WO2019136685A1 (en) | Method and apparatus for network selection by terminal, and computer device and storage medium | |
| CN108874441A (en) | A kind of board configuration method, device, server and storage medium | |
| WO2024139843A1 (en) | Cross-language video processing method and apparatus, and device, medium and product | |
| CN113760318B (en) | Information processing method, device, server and storage medium | |
| CN109347899A (en) | The method of daily record data is written in distributed memory system | |
| CN108845892A (en) | Data processing method, device, equipment and the computer storage medium of distributed data base | |
| CN107862035A (en) | Network reading method and device for conference record, intelligent tablet and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181116 |
|
| RJ01 | Rejection of invention patent application after publication |