CN113553379A - Data acquisition method and device - Google Patents
Data acquisition method and device Download PDFInfo
- Publication number
- CN113553379A CN113553379A CN202110833620.3A CN202110833620A CN113553379A CN 113553379 A CN113553379 A CN 113553379A CN 202110833620 A CN202110833620 A CN 202110833620A CN 113553379 A CN113553379 A CN 113553379A
- Authority
- CN
- China
- Prior art keywords
- data
- data acquisition
- database
- available information
- database synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 239000003795 chemical substances by application Substances 0.000 claims description 50
- 238000012360 testing method Methods 0.000 claims description 36
- 238000013480 data collection Methods 0.000 claims description 12
- 239000012634 fragment Substances 0.000 claims description 9
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 33
- 230000006870 function Effects 0.000 abstract description 10
- 230000002829 reductive effect Effects 0.000 abstract description 3
- 238000001514 detection method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000005055 memory storage Effects 0.000 description 2
- 230000036961 partial effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data acquisition method and device, and the system comprises a database synchronization component and a database synchronization agent based on a data acquisition system. When the data acquisition system in the specification is in an operating state, all ends related to the data acquisition process are not transparent according to codes inside the container assembly obtained by the database synchronization assembly, and for the container assembly, the functions of the container assembly can be realized in the data acquisition process only by configuring a data synchronization interface of the container assembly without adjusting the codes in the database synchronization assembly, so that the use threshold of the data acquisition system in the specification is effectively reduced.
Description
Technical Field
The application relates to the technical field of data acquisition, in particular to a data acquisition method and device.
Background
With the development of the information society, the processing of various services depends on the acquisition of data to a great extent.
The object targeted by the data acquisition process is called a database, taking a relational database as an example, the data storage modes of all databases are not completely the same, so that the data acquisition performed on the databases faces a more complex acquisition environment. In addition, data users have different requirements on data, and the complexity of data acquisition is increased to a certain extent. Therefore, a data user has to maintain and manage the database synchronization component adopted during data acquisition according to the service requirement of the data user. However, most of the database synchronization components are code-based applications or devices with applications solidified, and maintenance and management of the database synchronization components require strong professional knowledge, so that it is difficult for a user to know the codes of the database synchronization components, and a data acquisition threshold is increased.
Disclosure of Invention
The application provides a data acquisition method and a data acquisition device, which effectively reduce the use threshold of a data acquisition system, and adopts the following technical scheme:
in a first aspect, a data acquisition method is provided, the method is based on a data acquisition system, and the system comprises a database synchronization component and a database synchronization agent; the method comprises the following steps:
acquiring metadata of each preset database;
determining available information according to the metadata, wherein the available information shows the storage position of the data to be acquired in the database;
according to the available information, containerizing the database synchronization assembly to obtain a container assembly;
establishing a connection between the container component and the database synchronization agent according to the available information;
and adopting the connected container assembly and the database synchronous proxy to acquire data.
Optionally, before determining the available information according to the metadata, the method further includes:
determining description information of each data acquired by the data acquisition system in a preset time period according to each data acquisition request received by the data acquisition system in the preset time period;
determining available information from the metadata, including:
and determining data matched with the description information in the metadata as available information.
Optionally, according to the available information, containerizing the database synchronization component to obtain a container component, including:
processing according to the storage position of the data corresponding to the available information in the database to obtain a plurality of fragments;
according to the obtained fragments, the database synchronization component is containerized, so that the obtained container component comprises a plurality of data synchronization interfaces, and the data synchronization interfaces are in one-to-one correspondence with the fragments.
Optionally, establishing a connection between the container component and the database synchronization agent according to the available information includes:
and adding a data synchronization interface obtained according to the available information to the database synchronization agent so as to establish connection between the container component and the database synchronization agent.
Optionally, before performing data acquisition, the method further includes, by using the connected container component and the database synchronization agent:
generating a designated interface of the database synchronization agent according to the available information; the designated interface is used for acquiring a data acquisition request corresponding to the available information.
Optionally, the data acquisition is performed by using the connected container component and the database synchronization agent, and the data acquisition includes:
when a data acquisition request is detected, determining whether the data acquisition request is matched with the specified interface;
if so, enabling the database synchronization agent to generate a data acquisition instruction according to the data acquisition request, and sending the data acquisition instruction to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.
Optionally, before performing data acquisition, the method further includes, by using the connected container component and the database synchronization agent:
generating a first test request according to the available information, and generating a second test request according to information except the available information;
inputting the first test request and the second test request into the specified interface;
when the state of the data acquired according to the first test request is normal and the state of the data acquired according to the second test request is abnormal, judging that the state of the data acquisition system is normal;
adopting the connected container assembly and the database synchronous proxy to carry out data acquisition, comprising:
and under the condition that the state of the data acquisition system is normal, acquiring data.
In a second aspect, a data acquisition device is provided, wherein the device is used in a data acquisition system, and the system comprises a database synchronization component and a database synchronization agent; the device comprises:
the metadata acquisition module is configured to acquire metadata of each preset database;
the available information determining module is configured to determine available information according to the metadata, wherein the available information shows the storage position of the data to be acquired in the database;
the container component generation module is configured to containerize the database synchronization component according to the available information to obtain a container component;
a connection module configured to establish a connection between the container component and the database synchronization agent according to the availability information;
and the acquisition module is configured to adopt the connected container assembly and the database synchronous proxy to acquire data.
Optionally, the data acquisition system may further include a description information obtaining module 310.
The description information acquisition module is configured to determine description information of each data acquired by the data acquisition system in a preset time period according to each data acquisition request received by the data acquisition system in the preset time period;
and the available information determining module is specifically configured to determine data matched with the description information in the metadata as available information.
Optionally, the container component generation module may include a fragmentation sub-module and a data synchronization interface determination sub-module.
Optionally, the connection module is specifically configured to add a data synchronization interface obtained according to the available information to the database synchronization agent, so as to establish a connection between the container component and the database synchronization agent.
Optionally, the data acquisition system may further include a designated interface generation module. The specified interface generating module is configured to generate a specified interface of the database synchronization agent according to the available information; the designated interface is used for acquiring a data acquisition request corresponding to the available information.
Optionally, the acquisition module is specifically configured to determine whether the data acquisition request matches the specified interface when the data acquisition request is detected. If so, enabling the database synchronization agent to generate a data acquisition instruction according to the data acquisition request, and sending the data acquisition instruction to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.
Optionally, the data acquisition system may further include a detection module, where the detection module is configured to generate a first test request according to the available information, and generate a second test request according to information other than the available information. And inputting the first test request and the second test request into the specified interface. And when the state of the data acquired according to the first test request is normal and the state of the data acquired according to the second test request is abnormal, judging that the state of the data acquisition system is normal.
Optionally, the acquisition module is specifically configured to acquire data under the condition that the state of the data acquisition system is normal.
In a third aspect, an electronic device is provided, which includes:
one or more processors;
a memory;
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: the data acquisition method of the first aspect is performed.
In a fourth aspect, a computer-readable storage medium is provided, which is used for storing computer instructions, which when run on a computer, make the computer perform the data acquisition method shown in the first aspect.
The application provides a data acquisition method, a data acquisition device, electronic equipment and a computer-readable storage medium. When the data acquisition system in the specification is in an operating state, all ends related to the data acquisition process are not transparent according to codes inside the container assembly obtained by the database synchronization assembly, and for the container assembly, the functions of the container assembly can be realized in the data acquisition process only by configuring a data synchronization interface of the container assembly without adjusting the codes in the database synchronization assembly, so that the use threshold of the data acquisition system in the specification is effectively reduced.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic diagram of a sub-data acquisition system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a data collection process according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data acquisition device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
In the description of the embodiments of the present application, the words "exemplary," "for example," or "for instance" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary," "e.g.," or "e.g.," is not to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary," "e.g.," or "exemplary" is intended to present relevant concepts in a concrete fashion.
In the description of the embodiments of the present application, the term "and/or" is only one kind of association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, B exists alone, and A and B exist at the same time. In addition, the term "plurality" means two or more unless otherwise specified. For example, the plurality of systems refers to two or more systems, and the plurality of terminals refers to two or more terminals.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicit indication of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present description may be combined with each other without conflict.
Flow charts are used in this description to illustrate operations performed by a system according to embodiments of the present description. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.
In an actual data collection scenario, a data user often faces a more complex data collection environment. In order to match with the service requirements of users, which may change, and different service requirements of different users, so that the effect of data acquisition can reach the expectation, users often need to maintain and manage the codes of the database synchronization component according to the service requirements, which results in higher difficulty of data acquisition.
In view of this, the data acquisition process in this specification is presented. The data acquisition process in this specification is based on a data acquisition system. Illustratively, as shown in fig. 1, the data collection system includes a data source database synchronization component and a data source database synchronization agent, and the data collection system is configured to collect data from a data source database 1 to a data source database n, and then push the collected data to the distributed publish-subscribe messaging system, so that a user can use the collected data.
As shown in fig. 2, the process may include one or more of the following steps.
S200: and acquiring metadata of each preset data source database.
The database (Data Base) in this specification refers to a warehouse in which Data is stored. Is an organized, sharable, uniformly managed collection of large amounts of data that is stored long term within a computer. The type of database involved is not limited by the present description, and in alternative embodiments, at least a portion of the database coupled to the data acquisition system of the present description may store data in the form of tables. For example, the database in this specification may be a relational database.
In this specification, Metadata (also called intermediate Data or relay Data) is Data (Data About Data) describing Data, and is mainly information describing Data attribute (Property) for supporting functions such as indicating storage location, history Data, resource search, file record, and the like. The metadata has the function of an electronic catalog.
S202: and determining available information according to the metadata, wherein the available information shows the storage position of the data to be acquired in the database.
After the metadata of each data source database is obtained, the metadata obtained by different data source databases may be selected, so as to determine, for each data source database, information such as a storage location, a data identifier, and the like of data in the data source database, which is available for the data acquisition system in this specification, in the data source database, as available information.
It can be seen that the available information in this specification is used to obtain data corresponding to the available information from the database in the subsequent step, and besides, the available information in this specification can show the storage location of the data to be acquired in the database, the content included in the available information can be designed according to the actual requirement. For example, description information (for example, the aforementioned data attribute) of data corresponding to the availability information may be used as the availability information.
For example, in an alternative business scenario, the data to be collected is a record of the usage of the user's bank account, such as transaction amount, transaction time, and the like. The storage location of data stored in a database indicating usage records of bank accounts may be used as information available in this specification. Meanwhile, the database also stores information representing user attributes, such as the age, occupation and the like of the user, but the data are not the collected objects, and the information such as the storage positions of the data is not available information in the specification.
S204: and according to the available information, containerizing the database synchronization assembly to obtain a container assembly.
The database synchronization component is packaged, so that the database synchronization component only exposes a plurality of data synchronization interfaces, when the data acquisition system in the specification is in an operating state, all ends related to the data acquisition process are not transparent according to codes inside the database synchronization component, and for the container component, the functions of the container component can be realized in the data acquisition process only by configuring the data synchronization interfaces without adjusting the codes in the database synchronization component, so that the use threshold of the data acquisition system in the specification is effectively reduced.
S206: and establishing connection between the container component and the database synchronization agent according to the available information.
The process of establishing a connection is the process of associating the container component to the database synchronization agent. When data acquisition is performed through the data acquisition system in the specification, a user of the data can perform personalized setting on the container assembly through the database synchronous proxy, so that the user can conveniently control and manage the data acquisition process.
As described above, the containerization process of the database synchronization component is based on the available information, and when the connection between the container component and the database synchronization agent is established, the connection effect can be effectively ensured by still using the available information as the basis.
For example, in an alternative embodiment, the data synchronization interface of the container component is determined by the database identifier in the available information (which can roughly show the location of the data to be collected), and then the database synchronization agent may establish a connection between the data synchronization interface and the database synchronization agent for each data synchronization interface of the container component according to the database identifier in the available information.
S208: and adopting the connected container assembly and the database synchronous proxy to acquire data.
After the connection between the container component and the database synchronization agent is established, the user can configure the container component through the database synchronization component and control the data acquisition process so as to achieve the purpose of data acquisition.
From the foregoing, the containerization process performed on the database synchronization component plays an important role in this specification. The process of how this containerization is achieved will now be described.
The containerization process may be: and processing according to the storage position of the data corresponding to the available information in the database to obtain a plurality of fragments. According to the obtained fragments, the database synchronization component is containerized, so that the obtained container component comprises a plurality of data synchronization interfaces, and the data synchronization interfaces are in one-to-one correspondence with the fragments.
Then, in the subsequent step, when the container component and the database synchronization agent are established, a data synchronization interface obtained according to the available information may be added to the database synchronization agent to establish a connection between the container component and the database synchronization agent.
After the container component and the database synchronization agent are established, the data acquisition system is obtained with a state of being available. The data acquisition system whose status is available may be used for data acquisition. The data acquisition process may be to receive a data acquisition request sent by a user, and perform data acquisition according to the data acquisition request.
The data collection process in this specification may be to perform data collection under the trigger of a data acquisition request. In order to receive a data acquisition request, the data acquisition system in this specification is provided with a designated interface. The designated interface is an interface corresponding to data that the data acquisition system can provide to a data user, the designated interface being for receiving a data acquisition request corresponding to the available data, provided that the data is available.
The process of generating the specified interface may be: generating a designated interface of the database synchronization agent according to the available information; the designated interface is used for acquiring a data acquisition request corresponding to the available information.
The data collection process after generating the specified interface from the available information may be: upon detecting a data fetch request, determining whether the data fetch request matches the specified interface (e.g., the process may be determining whether the data fetch request corresponds to available information);
if so, enabling the database synchronization agent to generate a data acquisition instruction according to the data acquisition request, and sending the data acquisition instruction to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction. If not, sending a prompt to a sender of the data acquisition request, so that the data acquisition request is sent to the regenerated data acquisition request. Optionally, the prompt may be the availability information, so that the sender may regenerate the data acquisition request based on the availability information.
In an alternative embodiment of the present specification, when the data acquisition request does not match the specified interface, the information with the highest similarity to the data acquisition request may be determined from the available information, and the determined information may be displayed to the sender.
In the foregoing embodiment, the transaction amount may be used as available data, the attribute of the user is not available data, and the specified interface may be used to receive a data acquisition request for the transaction amount, but not for the attribute of the user.
In an alternative embodiment of the present description, the designation interface may be in the form of an interface to facilitate manipulation by a user of the data. In particular, the interface may include graphical and/or textual presentation areas, text entry areas, buttons, and the like.
Further, the processes herein not only enable a data acquisition system to be obtained that can be conveniently used by a user of data, but also enable the data acquisition system to be tested to determine whether the data acquisition system is in a usable normal state.
Specifically, the process of detecting may be: generating a first test request according to the available information, and generating a second test request according to information except the available information; and inputting the first test request and the second test request into the specified interface. And when the state of the data acquired according to the first test request is normal and the state of the data acquired according to the second test request is abnormal, judging that the state of the data acquisition system is normal.
The first test request is generated according to the available information and has a property of positive example, and if the data acquisition system is normal, data can be acquired according to the first test request. The second test request is obtained by information except the available information, and has the property of negative example, if the data acquisition system is normal, data cannot be acquired according to the second test request.
If the data acquisition system does not acquire data according to the first test request or the data acquisition system acquires data according to the second test request, it is indicated that some functions of the data acquisition system are abnormal.
After determining that the data acquisition system is abnormal, steps S204 and beyond may be re-executed.
Alternatively, the aforementioned detection of the data collection system may be performed after each establishment of a connection between the container component and the database synchronization agent.
As can be seen from the foregoing, the available information plays a more critical role in the data acquisition process of the present specification, and in an alternative embodiment of the present specification, the process of determining the available information may be: according to each data acquisition request received by the data acquisition system in a historical preset time period, determining the description information of each data acquired by the data acquisition system in the preset time period. And then, determining data matched with the description information in the metadata as available information.
The data obtaining request may be sent by a user of the data. The data acquisition system in the present specification may be oriented to a plurality of data users, for example, a data user may be a plurality of different companies, and each company may be a data user; for another example, the data consumers may be different departments of the same company, and any one of the departments may be a data consumer.
In order to meet the requirements of different users, when the preset time period is determined, the preset time period can cover at least one data acquisition request sent by each user related to the data acquisition system.
In an alternative embodiment of the present disclosure, the predetermined time period may be determined according to the degree of data demand of different users. If the higher the user's requirement level for data indicates that the higher the user's requirement level for data, the weight of the user may be determined for each user, and the weight may be determined according to a preset business rule. For each user, receiving a data acquisition request sent by the user, accumulating the number of times of the data acquisition request sent by each user according to the weight of each user (for example, performing weighted summation on the number of times of the data acquisition request sent by each user), and determining available information if the accumulated result reaches a preset threshold, wherein at this time, the preset time period is dynamically changed according to the accumulated result.
It should be noted that, in this embodiment, the data acquisition request used when determining the available information is not only used for determining the available information, but when the data acquisition request is received in the history, corresponding data acquisition processing is performed according to the data acquisition system before determining the available information. It can be seen that the process of determining available information in this specification may be regarded as an intermittent process of resetting the data acquisition system.
The specification does not limit the specific type of the data source database synchronization component, and in an alternative embodiment, the data source database synchronization component may be a canal component and the distributed publish-subscribe message system may be kafka. The data acquisition system may be a distributed data acquisition system. The platform adopted by the data acquisition system can be based on an OpenShift platform.
The number of database synchronization components and the number of database synchronization agents included in the data acquisition system are not limited in this specification.
The present specification further provides a data acquisition device, which can execute a data acquisition process provided in the above embodiments of the present application, as shown in fig. 3, the data acquisition device is applied to a data acquisition system; the data acquisition system comprises a data source database synchronization component and a data source database synchronization agent, and the device comprises:
a metadata obtaining module 300 configured to obtain, for each preset data source database, metadata of the data source database;
an available information determination module 302 configured to determine available information according to the metadata, wherein the available information shows a storage position of the data to be acquired in the database;
a container component generation module 304, configured to perform containerization on the database synchronization component according to the available information to obtain a container component;
a connection module 306 configured to establish a connection between the container component and the database synchronization agent according to the availability information;
and the acquisition module 308 is configured to perform data acquisition by using the connected container component and the database synchronization agent.
Optionally, the data acquisition system may further include a description information obtaining module 310.
A description information obtaining module 310, configured to determine, according to each data obtaining request received by the data collecting system within a historical preset time period, description information of each data collected by the data collecting system within the preset time period;
the available information determining module 302 is specifically configured to determine, as available information, data matching the description information in the metadata.
Optionally, the container component generation module 304 may include a fragmentation submodule 3040 and a data synchronization interface determination submodule 3042.
Optionally, the connection module 306 is specifically configured to add a data synchronization interface obtained according to the available information to the database synchronization agent, so as to establish a connection between the container component and the database synchronization agent.
Optionally, the data acquisition system may further include a designated interface generation module 312. The designated interface generation module 312 is configured to generate a designated interface of the database synchronization agent according to the available information; the designated interface is used for acquiring a data acquisition request corresponding to the available information.
Optionally, the acquiring module 308 is specifically configured to determine whether the data obtaining request matches the specified interface when the data obtaining request is detected. If so, enabling the database synchronization agent to generate a data acquisition instruction according to the data acquisition request, and sending the data acquisition instruction to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.
Optionally, the data acquisition system may further include a detection module 314, where the detection module 314 is configured to generate a first test request according to the available information, and generate a second test request according to information other than the available information. And inputting the first test request and the second test request into the specified interface. And when the state of the data acquired according to the first test request is normal and the state of the data acquired according to the second test request is abnormal, judging that the state of the data acquisition system is normal.
Optionally, the acquisition module 308 is specifically configured to perform data acquisition under the condition that the state of the data acquisition system is normal.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of one or more embodiments of the present specification. One of ordinary skill in the art can understand and implement it without inventive effort.
Further, an embodiment of the present application provides an electronic device, as shown in fig. 4, where the electronic device 40 shown in fig. 4 includes: a processor 401 and a memory storage device 403. Wherein the processor 401 is coupled to a memory storage device 403, such as via a bus 402. Further, the electronic device 40 may also include a transceiver 404. It should be noted that the transceiver 404 is not limited to one in practical applications, and the structure of the electronic device 40 is not limited to the embodiment of the present application. The processor 401 is applied to the embodiment of the present application, and is used to implement the functions of the respective modules shown in fig. 3. The transceiver 404 includes a receiver and a transmitter.
The processor 401 may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 401 may also be a combination of computing functions, e.g., comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
The storage device 403 is used for storing application program codes for executing the scheme of the application, and the execution is controlled by the processor 401. The processor 401 is configured to execute application program code stored in the storage device 403 for implementing the functions of the various modules shown in fig. 3.
The present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method shown in the above embodiments is implemented.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.
Claims (10)
1. A data acquisition method is based on a data acquisition system, and the system comprises a database synchronization component and a database synchronization agent; the method comprises the following steps:
acquiring metadata of each preset database;
determining available information according to the metadata, wherein the available information shows the storage position of the data to be acquired in the database;
according to the available information, containerizing the database synchronization assembly to obtain a container assembly;
establishing a connection between the container component and the database synchronization agent according to the available information;
and adopting the connected container assembly and the database synchronous proxy to acquire data.
2. The method of claim 1, wherein prior to determining available information based on the metadata, the method further comprises:
determining description information of each data acquired by the data acquisition system in a preset time period according to each data acquisition request received by the data acquisition system in the preset time period;
determining available information from the metadata, including:
and determining data matched with the description information in the metadata as available information.
3. The method of claim 1, wherein containerizing the database synchronization component based on the availability information to obtain a container component comprises:
processing according to the storage position of the data corresponding to the available information in the database to obtain a plurality of fragments;
according to the obtained fragments, the database synchronization component is containerized, so that the obtained container component comprises a plurality of data synchronization interfaces, and the data synchronization interfaces are in one-to-one correspondence with the fragments.
4. The method of claim 3, wherein establishing a connection between the container component and the database synchronization agent based on the availability information comprises:
and adding a data synchronization interface obtained according to the available information to the database synchronization agent so as to establish connection between the container component and the database synchronization agent.
5. The method of claim 1, wherein prior to performing data collection using the connected container component and the database synchronization agent, the method further comprises:
generating a designated interface of the database synchronization agent according to the available information; the designated interface is used for acquiring a data acquisition request corresponding to the available information.
6. The method of claim 5, wherein using the connected container component and the database synchronization agent for data collection comprises:
when a data acquisition request is detected, determining whether the data acquisition request is matched with the specified interface;
if so, enabling the database synchronization agent to generate a data acquisition instruction according to the data acquisition request, and sending the data acquisition instruction to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.
7. The method of claim 6, wherein prior to data collection using the connected container component and the database synchronization agent, the method further comprises:
generating a first test request according to the available information, and generating a second test request according to information except the available information;
inputting the first test request and the second test request into the specified interface;
when the state of the data acquired according to the first test request is normal and the state of the data acquired according to the second test request is abnormal, judging that the state of the data acquisition system is normal;
adopting the connected container assembly and the database synchronous proxy to carry out data acquisition, comprising:
and under the condition that the state of the data acquisition system is normal, acquiring data.
8. A data acquisition device, wherein the device is used in a data acquisition system, the system comprising a database synchronization component and a database synchronization agent; the device comprises:
the metadata acquisition module is configured to acquire metadata of each preset database;
the available information determining module is configured to determine available information according to the metadata, wherein the available information shows the storage position of the data to be acquired in the database;
the container component generation module is configured to containerize the database synchronization component according to the available information to obtain a container component;
a connection module configured to establish a connection between the container component and the database synchronization agent according to the availability information;
and the acquisition module is configured to adopt the connected container assembly and the database synchronous proxy to acquire data.
9. An electronic device, one or more processors;
a memory;
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: performing the method of any of the preceding claims 1-7.
10. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110833620.3A CN113553379A (en) | 2021-07-23 | 2021-07-23 | Data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110833620.3A CN113553379A (en) | 2021-07-23 | 2021-07-23 | Data acquisition method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113553379A true CN113553379A (en) | 2021-10-26 |
Family
ID=78104225
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110833620.3A Pending CN113553379A (en) | 2021-07-23 | 2021-07-23 | Data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113553379A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114567633A (en) * | 2022-03-01 | 2022-05-31 | 上海浦东发展银行股份有限公司 | Cloud platform system supporting full life cycle of multi-stack database and management method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207894A (en) * | 2013-03-14 | 2013-07-17 | 深圳市知正科技有限公司 | Multipath real-time video data storage system and cache control method thereof |
| CN112395363A (en) * | 2020-12-04 | 2021-02-23 | 北京有竹居网络技术有限公司 | Data synchronization method, device, equipment and readable storage medium |
-
2021
- 2021-07-23 CN CN202110833620.3A patent/CN113553379A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207894A (en) * | 2013-03-14 | 2013-07-17 | 深圳市知正科技有限公司 | Multipath real-time video data storage system and cache control method thereof |
| CN112395363A (en) * | 2020-12-04 | 2021-02-23 | 北京有竹居网络技术有限公司 | Data synchronization method, device, equipment and readable storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114567633A (en) * | 2022-03-01 | 2022-05-31 | 上海浦东发展银行股份有限公司 | Cloud platform system supporting full life cycle of multi-stack database and management method |
| CN114567633B (en) * | 2022-03-01 | 2024-06-28 | 上海浦东发展银行股份有限公司 | Cloud platform system supporting full life cycle of multi-stack database and management method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107798108B (en) | Asynchronous task query method and device | |
| KR101805383B1 (en) | Launching a client application based on a message | |
| CN107846321B (en) | Interface monitoring method and device and electronic equipment | |
| CN109213519B (en) | Heterogeneous API adaptation processing method and system | |
| CN107276970B (en) | Unbinding and binding method and device | |
| CN112055016B (en) | Message generation method and device, electronic equipment and storage medium | |
| US20220052976A1 (en) | Answer text processing methods and apparatuses, and key text determination methods | |
| CN113435862B (en) | Bill processing method and device based on mailbox | |
| CN110019444B (en) | Operation request processing method, device, equipment and system | |
| CN108089972B (en) | Interface test method and device | |
| CN113553379A (en) | Data acquisition method and device | |
| CN107633080B (en) | User task processing method and device | |
| CN111651522B (en) | Data synchronization method and device | |
| CN111488373B (en) | Method and system for processing request | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN111309576A (en) | Pressure testing method and device | |
| CN116821119A (en) | Method, device and server for generating message analysis code | |
| CN110083525A (en) | Localization method, device, computer equipment and the storage medium of RAM leakage | |
| CN116955148A (en) | Service system testing method, device, equipment, storage medium and product | |
| CN111967767A (en) | Business risk identification method, device, equipment and medium | |
| CN110442572B (en) | User characteristic value determining method and device | |
| CN113961632A (en) | Message data processing method and device and electronic equipment | |
| CN114386035A (en) | Method and device for detecting threat data and electronic equipment | |
| CN112181825A (en) | Test case library construction method and device, electronic equipment and medium | |
| CN112597119A (en) | Method and device for generating processing log and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |