CN110119422B - Data processing system and equipment for data warehouse of micro-credit tenant - Google Patents
Data processing system and equipment for data warehouse of micro-credit tenant Download PDFInfo
- Publication number
- CN110119422B CN110119422B CN201910405548.7A CN201910405548A CN110119422B CN 110119422 B CN110119422 B CN 110119422B CN 201910405548 A CN201910405548 A CN 201910405548A CN 110119422 B CN110119422 B CN 110119422B
- Authority
- CN
- China
- Prior art keywords
- data
- tenant
- layer
- etl
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 66
- 238000007726 management method Methods 0.000 claims abstract description 42
- 238000000605 extraction Methods 0.000 claims abstract description 33
- 239000000872 buffer Substances 0.000 claims abstract description 20
- 238000000926 separation method Methods 0.000 claims abstract description 18
- 230000004931 aggregating effect Effects 0.000 claims abstract description 9
- 238000013468 resource allocation Methods 0.000 claims abstract description 9
- 238000000034 method Methods 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 11
- 230000002776 aggregation Effects 0.000 claims description 9
- 238000004220 aggregation Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 7
- 238000011068 loading method Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000013075 data extraction Methods 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 4
- 238000011144 upstream manufacturing Methods 0.000 claims description 4
- 238000012384 transportation and delivery Methods 0.000 claims 1
- 239000000284 extract Substances 0.000 abstract description 3
- 230000001360 synchronised effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/03—Credit; Loans; Processing thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data processing system and equipment for a data warehouse of a minor credit tenant. The system comprises: the global configuration layer is connected with the master control management layer and configures global information; the tenant configuration layer is connected with the master control management layer and is used for configuring information for the tenant; the master control management layer is connected with the tenant data pool, and issues interface tasks and coordinates resource allocation; the tenant data pool is connected with the ETL tenant extraction layer, a memory management area is distributed to the tenant, interface tasks are synchronized from the master control management layer, and/or an ETL interface timestamp and a table pool are initialized; the ETL tenant extraction layer is connected with the data warehouse data merging layer and extracts tenant data to the data buffer area; the data warehouse data merging layer is connected with the data interface separating layer and used for aggregating tenant data; the data interface separation layer is connected with the data authority control layer, so that online analysis processing of the OLAP query tenant data is realized; and the data authority control layer sets the tenant authority. The invention can ensure the stability, safety and high efficiency of data processing of the data warehouse.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data processing system and equipment for a data warehouse of a minor credit tenant.
Background
At present, with the development of the mini credit business, the number of tenants is increased, and the data volume is obviously increased. The design of the data warehouse needs to ensure the independence, timeliness, data fault tolerance, data backtracking and data diagnosis of data processing of a single tenant. Data security access is of great importance to enterprises, business system data are concentrated on one entity, and a data warehouse must be processed in a tenant data separation mode to ensure that each tenant operates the operation requirement of quick decision. Data security is also important, and if one tenant acquires the customer information of another tenant, economic loss of an enterprise cannot be evaluated. Meanwhile, Data processing tasks cannot be separated well, so that a plurality of tasks access the same table, and the interlocking and deadlock probability of DML (Data manipulation language) statements is increased, so that the stability and the Data processing efficiency of the database are affected. Therefore, finding a method for efficiently, stably and safely processing data in a data warehouse is an urgent technical problem to be solved in the industry.
Disclosure of Invention
In order to solve the above problems in the prior art, embodiments of the present invention provide a data processing system and device for a data warehouse of a minor credit tenant.
In a first aspect, an embodiment of the present invention provides a data processing system for a microcarty tenant data warehouse, including: the system comprises a global configuration layer, a tenant configuration layer, a master control management layer, a tenant data pool, an ETL tenant extraction layer, a data warehouse data merging layer, a data interface separation layer and a data authority control layer;
the global configuration layer is connected with the master control management layer and is used for configuring global information; the tenant configuration layer is connected with the master control management layer and is used for configuring information for tenants; the master control management layer is connected with the tenant data pool and used for dynamically issuing interface tasks and dynamically coordinating resource allocation; the tenant data pool is connected with the ETL tenant extraction layer and used for distributing a memory management area to tenants, synchronizing interface tasks from the master control management layer and/or initializing ETL interface timestamps and a table pool; the ETL tenant extraction layer is connected with the data warehouse data merging layer and used for extracting tenant data to a data buffer area; the data warehouse data merging layer is connected with the data interface separation layer and used for aggregating tenant data; the data interface separation layer is connected with the data authority control layer and used for enabling the OLAP to query the tenant data in an online analysis and processing mode; the data authority control layer is used for setting the authority of the tenant; wherein the ETL is a process of data extraction, conversion and loading.
Further, on the basis of the foregoing system embodiment, the system for data processing of a microcartry tenant data warehouse provided in an embodiment of the present invention, where the system for configuring global information includes: configuring configuration information of a host, a database and/or a file transfer protocol for exchanging data with a data warehouse; configuring parameter information of a master control manager; and configuring an extraction container operation template and copying an ETL data stream.
Further, on the basis of the foregoing system embodiment, the system for data processing of a microcartry tenant data warehouse provided in the embodiment of the present invention, which is used for configuring information for a tenant, includes: and configuring basic information of the tenant, resource information of the tenant and ETL interface information.
Further, on the basis of the foregoing system embodiment, the system for processing a microcartry tenant data warehouse according to an embodiment of the present invention is configured to dynamically issue interface tasks, where the system includes: and triggering the workflow processed by the ETL interface according to the ETL interface information.
Further, on the basis of the above system embodiment, the system for data warehouse data processing of microcartry tenants provided in the embodiment of the present invention for dynamically coordinating resource allocation includes: the resource usage status is obtained from an upstream business repository system and/or the physical resource usage status is collected from a data warehouse host.
Further, on the basis of the foregoing system embodiment, the system for data processing of a microcartry tenant data warehouse provided in the embodiment of the present invention, where the extracting tenant data into a data buffer includes: and acquiring a database connection instance and an ETL time stamp from the tenant data pool, calling an ETL packet, and extracting tenant data to a data buffer area.
Further, on the basis of the foregoing system embodiment, the system for processing a microcarty tenant data warehouse data provided in an embodiment of the present invention, which is used for aggregating tenant data, includes: and operating the data storage layer to perform parallel processing on tenant data aggregation, and applying a merging task lock for the tenant data when the tenant data aggregation is completed and then merging the data storage table.
In a second aspect, an embodiment of the present invention provides an electronic device, including:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor to invoke the program instructions to be able to execute the microcartry tenant data warehouse data processing system provided by any of the various possible implementations of the first aspect.
In a third aspect, embodiments of the invention provide a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute a microcapiuary tenant data repository data processing system as provided in any of the various possible implementations of the first aspect.
According to the data processing system and the data processing equipment for the minor credit tenant data warehouse, provided by the embodiment of the invention, the tenant data is extracted by adopting dynamic management and stored in the data buffer area, then the tenant data is merged at the data merging layer of the data warehouse, and finally the tenant data is inquired by using on-line analysis processing OLAP, so that the stability, the safety and the high efficiency of data processing in the data warehouse can be ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description will be given below to the drawings required for the description of the embodiments or the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic structural diagram of a data processing system of a microcarty tenant data warehouse according to an embodiment of the present invention;
fig. 2 is a schematic physical structure diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. In addition, technical features of various embodiments or individual embodiments provided by the invention can be arbitrarily combined with each other to form a feasible technical solution, but must be realized by a person skilled in the art, and when the technical solution combination is contradictory or cannot be realized, the technical solution combination is not considered to exist and is not within the protection scope of the present invention.
An embodiment of the present invention provides a data processing system for a data warehouse of a minor credit tenant, and referring to fig. 1, the system includes: the system comprises a global configuration layer, a tenant configuration layer, a master control management layer, a tenant data pool, an ETL tenant extraction layer, a data warehouse data merging layer, a data interface separation layer and a data authority control layer;
the global configuration layer is connected with the master control management layer and is used for configuring global information; the tenant configuration layer is connected with the master control management layer and is used for configuring information for tenants; the master control management layer is connected with the tenant data pool and used for dynamically issuing interface tasks and dynamically coordinating resource allocation; the tenant data pool is connected with the ETL tenant extraction layer and used for distributing a memory management area to tenants, synchronizing interface tasks from the master control management layer and/or initializing ETL interface timestamps and a table pool; specifically, the tenant separately allocates a memory management area, initializes a CONNECT POOL, and synchronizes the number of connections from the total control management layer. And synchronizing the interface TASK from the master control management layer and initializing the TASK POOL. The extraction time stamp of the ETL interface configured by the tenant is initialized, and the TABLPOOL (table pool) is initialized, so that the extraction time stamp can be conveniently acquired when the extraction task is executed. The ETL tenant extraction layer is connected with the data warehouse data merging layer and used for extracting tenant data to a data buffer area; specifically, the ETL tenant extraction layer extracts three tenants (ETL-a tenants, ETL-B tenants, and ETL-C tenants, respectively) and stores the three tenants in three data buffers (data buffer a, data buffer B, and data buffer C, respectively). The data warehouse data merging layer is connected with the data interface separation layer and used for aggregating tenant data; the data interface separation layer is connected with the data authority control layer and used for enabling the OLAP to query the tenant data in an online analysis and processing mode; specifically, the data interface separation layer in this embodiment includes: and the cube comprises cube A, cube B and cube C, and the cube A, cube B and cube C respectively store the data of ETL-A tenant, ETL-B tenant and ETL-C tenant. The data authority control layer is used for setting the authority of the tenant; specifically, the set tenant permissions include: and the network access user authentication is mainly used for detecting the user name and the password authentication of the login of the tenant. Effectively carrying out the next verification; database security authentication, which is mainly used for detecting whether a user name logged in by a tenant has a bound database user at a database end, and if so, entering the next step; and controlling the data authority range, namely mainly detecting that the ID of the login USER corresponds to the USER _ ID one by one, establishing a relation between the authority control table corresponding to the USER _ ID and other dimension and degree tables, injecting the USER _ ID into a query where statement, and limiting a return result in the authority range of the tenant. Wherein the ETL is a process of data extraction, conversion and loading. The number of the tenants in this embodiment is three, and the three tenants are tenant a, tenant B, and tenant C. It should be noted that the number of tenants may be several, for example, 1, 2 or 5, and the number is determined by the actual operating state of the system, and is not limited to the specific number mentioned above. Specifically, a tenant refers to a business or organization that provides a small amount of credit (e.g., a small loan), and each such business, organization, or individual in the data warehouse data processing system of the small amount of credit tenant in the embodiment of the present invention is referred to as a tenant. In order to optimize the data processing flow of the data warehouse data processing system, the master control management layer can also call an application program from the API (application program interface) of the service library to dynamically update and extract parameters; and an application program can be called from the API interface of the local server to realize dynamic updating of the loading parameters.
On the basis of the foregoing embodiment, the system for data processing of a microcartry tenant data warehouse, provided in an embodiment of the present invention, for configuring global information includes: configuring configuration information of a host, a database and/or a file transfer protocol for exchanging data with a data warehouse; configuring parameter information of a master control manager; and configuring an extraction container operation template and copying an ETL data stream. Specifically, configuration information of HOST, DATABASE, FTP (file transfer protocol) for exchanging data with the data warehouse is configured. Configuring parameter information of a master control management layer, comprising the following steps: the method comprises the steps of parallelly processing the maximum number of tenants, the ETL extraction task number, the data warehouse model collection task number, the multidimensional data cube aggregation task number and the network connection number. And configuring an extraction container operation template, and quickly copying an ETL data stream. And then according to the flow of reading a timestamp, creating a temporary table, emptying temporary data, loading extraction data, a data conversion rule, warehousing and collecting statistical information of the table, and completing the operation template of the configuration extraction container.
On the basis of the above embodiments, the system for data processing of a data warehouse of a minor credit tenant provided in an embodiment of the present invention is configured to configure information for the tenant, and includes: and configuring basic information of the tenant, resource information of the tenant and ETL interface information. Specifically, the tenant basic information includes: tenant ID, tenant name, tenant priority, temporary buffer for storage, database for storage, tablespace, naming convention information. And configuring tenant resource information and ETL interface information, and referencing resources of the global configuration area. The ETL interface information comprises: interface number, temporary table name, SQL extraction and interface operation time.
On the basis of the foregoing embodiment, the system for processing a data warehouse of a minor credit tenant provided in the embodiment of the present invention is configured to dynamically issue interface tasks, and includes: and triggering the workflow processed by the ETL interface according to the ETL interface information. Specifically, according to ETL interface information configured by a tenant, an interface is triggered at regular time to process a workflow, and the working state of each process is recorded. If a certain node fails, the task can be retransmitted from the breakpoint.
On the basis of the foregoing embodiments, the data processing system for a microcartty tenant data warehouse provided in an embodiment of the present invention, which is used for dynamically coordinating resource allocation, includes: the resource usage status is obtained from an upstream business repository system and/or the physical resource usage status is collected from a data warehouse host. Specifically, controlling the quota of tenant data pool resources, in order to ensure that the extraction does not affect the performance of the service database, acquiring the resource use state from the upstream service database system every 15 minutes, includes: the input and output interface, the central processing unit, the database connection number, the table space and the transaction log state adjust the maximum allowable extraction task number in real time. And the system physical resource use state can be collected from each host computer of the data warehouse so as to adjust the number of the parallel execution tasks of the nodes behind the extraction task.
On the basis of the foregoing embodiment, the system for data processing of a minor credit tenant data warehouse, provided in an embodiment of the present invention, where the extracting tenant data into a data buffer includes: and acquiring a database connection instance and an ETL time stamp from the tenant data pool, calling an ETL packet, and extracting tenant data to a data buffer area. Specifically, the extracting step includes: acquiring a database connection INSTANCE INSTANCE _ ID from a tenant data pool; acquiring an ETL timestamp from a tenant data pool; and calling the ETL packet, extracting all tenant data to a temporary buffer area, and merging and summarizing the tenant data to a data warehouse ODS (operation data storage) layer.
On the basis of the above embodiments, the system for data processing of a microcarty tenant data warehouse provided in an embodiment of the present invention is used for aggregating tenant data, and includes: and operating the data storage layer to perform parallel processing on tenant data aggregation, and applying a merging task lock for the tenant data when the tenant data aggregation is completed and then merging the data storage table. Specifically, the data warehouse data is a collection of tenants, in order to ensure that each tenant does not need to wait for other tenants when analyzing the data, the ODS layer performs parallel processing on tenant data aggregation, when merging the data warehouse table after each tenant data aggregation is completed, a merging task lock is applied, different from the database lock, the database lock ensures the consistency and the integrity of the data, and under the condition that multiple tenants insert the same table in large data volume, the lock usually waits, and then deadlock is possibly caused, and the merging task lock can enable other tenants to be in a merging waiting state.
On the basis of the above embodiments, the system for processing a microcach tenant data warehouse data, provided in an embodiment of the present invention, for enabling online analytical processing OLAP query tenant data includes: according to the tenant ID, the online analysis and processing OLAP separates the tenant data from the data warehouse and stores the tenant data in the multidimensional data cube, and the online analysis and processing OLAP queries the tenant data from the multidimensional data cube. Specifically, in order to ensure the client access efficiency of the front-end network, an OLAP (Online Analytical Processing) layer separates tenant data in a data warehouse summary layer according to a tenant ID, and one tenant and one Cube ensure that data accessed by open tenants are isolated, so that the data access speed of the tenants is improved.
According to the data processing system for the minor credit tenant data warehouse provided by the embodiment of the invention, the tenant data is extracted by adopting dynamic management and stored in the data buffer area, then the tenant data is merged at the data merging layer of the data warehouse, and finally the tenant data is inquired by using on-line analysis processing OLAP, so that the stability, the safety and the high efficiency of data processing in the data warehouse can be ensured.
The data processing system of the data warehouse of the minor credit tenant in the embodiment of the invention is realized by relying on electronic equipment, so that the related electronic equipment is introduced. To this end, an embodiment of the present invention provides an electronic apparatus, as shown in fig. 2, including: at least one processor (processor)201, a communication Interface (communication Interface)204, at least one memory (memory)202 and a communication bus 203, wherein the at least one processor 201, the communication Interface 204 and the at least one memory 202 are configured to communicate with each other via the communication bus 203. The at least one processor 201 may invoke logic instructions in the at least one memory 202 to implement the following system: the system comprises a global configuration layer, a tenant configuration layer, a master control management layer, a tenant data pool, an ETL tenant extraction layer, a data warehouse data merging layer, a data interface separation layer and a data authority control layer; the global configuration layer is connected with the master control management layer and is used for configuring global information; the tenant configuration layer is connected with the master control management layer and is used for configuring information for tenants; the master control management layer is connected with the tenant data pool and used for dynamically issuing interface tasks and dynamically coordinating resource allocation; the tenant data pool is connected with the ETL tenant extraction layer and used for distributing a memory management area to tenants, synchronizing interface tasks from the master control management layer and/or initializing ETL interface timestamps and a table pool; the ETL tenant extraction layer is connected with the data warehouse data merging layer and used for extracting tenant data to a data buffer area; the data warehouse data merging layer is connected with the data interface separation layer and used for aggregating tenant data; the data interface separation layer is connected with the data authority control layer and used for enabling the OLAP to query the tenant data in an online analysis and processing mode; the data authority control layer is used for setting the authority of the tenant; wherein the ETL is a process of data extraction, conversion and loading.
Furthermore, the logic instructions in the at least one memory 202 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be substantially implemented or contributed to by the prior art, or the technical solution may be implemented in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to implement all or part of the system according to the embodiments of the present invention. Examples include: the system comprises a global configuration layer, a tenant configuration layer, a master control management layer, a tenant data pool, an ETL tenant extraction layer, a data warehouse data merging layer, a data interface separation layer and a data authority control layer; the global configuration layer is connected with the master control management layer and is used for configuring global information; the tenant configuration layer is connected with the master control management layer and is used for configuring information for tenants; the master control management layer is connected with the tenant data pool and used for dynamically issuing interface tasks and dynamically coordinating resource allocation; the tenant data pool is connected with the ETL tenant extraction layer and used for distributing a memory management area to tenants, synchronizing interface tasks from the master control management layer and/or initializing ETL interface timestamps and a table pool; the ETL tenant extraction layer is connected with the data warehouse data merging layer and used for extracting tenant data to a data buffer area; the data warehouse data merging layer is connected with the data interface separation layer and used for aggregating tenant data; the data interface separation layer is connected with the data authority control layer and used for enabling the OLAP to query the tenant data in an online analysis and processing mode; the data authority control layer is used for setting the authority of the tenant; wherein the ETL is a process of data extraction, conversion and loading. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. Based on this recognition, each block in the flowchart or block diagrams may represent a module, a program segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In this patent, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (9)
1. A microcartridge tenant data warehouse data processing system, comprising:
the system comprises a global configuration layer, a tenant configuration layer, a master control management layer, a tenant data pool, an ETL tenant extraction layer, a data warehouse data merging layer, a data interface separation layer and a data authority control layer;
the global configuration layer is connected with the master control management layer and is used for configuring global information;
the tenant configuration layer is connected with the master control management layer and is used for configuring information for tenants;
the master control management layer is connected with the tenant data pool and used for dynamically issuing interface tasks and dynamically coordinating resource allocation;
the tenant data pool is connected with the ETL tenant extraction layer and used for distributing a memory management area to tenants, synchronizing interface tasks from the master control management layer and/or initializing ETL interface timestamps and a table pool;
the ETL tenant extraction layer is connected with the data warehouse data merging layer and used for extracting tenant data to a data buffer area;
the data warehouse data merging layer is connected with the data interface separation layer and used for aggregating tenant data;
the data interface separation layer is connected with the data authority control layer and used for enabling the OLAP to query the tenant data in an online analysis and processing mode;
the data authority control layer is used for setting the authority of the tenant;
wherein the ETL is a process of data extraction, conversion and loading;
the method for enabling online analysis processing to process OLAP query tenant data comprises the following steps:
according to the tenant ID, the online analysis and processing OLAP separates the tenant data from the data warehouse and stores the tenant data in the multidimensional data cube, and the online analysis and processing OLAP queries the tenant data from the multidimensional data cube.
2. The microcartridge tenant data warehouse data processing system of claim 1, wherein the means for configuring global information comprises:
configuring configuration information of a host, a database and/or a file transfer protocol for exchanging data with a data warehouse;
configuring parameter information of a master control manager;
and configuring an extraction container operation template and copying an ETL data stream.
3. The microcartridge tenant data warehouse data processing system of claim 1, wherein the means for configuring information to tenants comprises:
and configuring basic information of the tenant, resource information of the tenant and ETL interface information.
4. The microcartridge tenant data warehouse data processing system of claim 3, wherein the interface tasks for dynamic delivery include:
and triggering the workflow processed by the ETL interface according to the ETL interface information.
5. The microcartridge tenant data warehouse data processing system of claim 1, wherein the means for dynamically coordinating resource allocation comprises:
the resource usage status is obtained from an upstream business repository system and/or the physical resource usage status is collected from a data warehouse host.
6. The microcartry tenant data warehouse data processing system of claim 1, wherein the extraction of tenant data into a data buffer comprises:
and acquiring a database connection instance and an ETL time stamp from the tenant data pool, calling an ETL packet, and extracting tenant data to a data buffer area.
7. The microcartridge tenant data warehouse data processing system of claim 1, wherein the means for aggregating tenant data comprises:
and operating the data storage layer to perform parallel processing on tenant data aggregation, and applying a merging task lock for the tenant data when the tenant data aggregation is completed and then merging the data storage table.
8. An electronic device, comprising:
at least one processor, at least one memory, a communication interface, and a bus; wherein,
the processor, the memory and the communication interface complete mutual communication through the bus;
the memory stores program instructions executable by the processor, which are invoked by the processor to implement the system of any one of claims 1 to 7.
9. A non-transitory computer-readable storage medium storing computer instructions that cause a computer to implement the system of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910405548.7A CN110119422B (en) | 2019-05-16 | 2019-05-16 | Data processing system and equipment for data warehouse of micro-credit tenant |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910405548.7A CN110119422B (en) | 2019-05-16 | 2019-05-16 | Data processing system and equipment for data warehouse of micro-credit tenant |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110119422A CN110119422A (en) | 2019-08-13 |
| CN110119422B true CN110119422B (en) | 2021-05-07 |
Family
ID=67522630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910405548.7A Active CN110119422B (en) | 2019-05-16 | 2019-05-16 | Data processing system and equipment for data warehouse of micro-credit tenant |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110119422B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111611252B (en) * | 2020-04-01 | 2023-07-18 | 石化盈科信息技术有限责任公司 | Monitoring, device, equipment and storage medium for safety data in data synchronization process |
| CN111552730B (en) * | 2020-04-28 | 2024-01-26 | 杭州数梦工场科技有限公司 | Data distribution method, device, electronic equipment and storage medium |
| US11641371B2 (en) * | 2021-02-17 | 2023-05-02 | Saudi Arabian Oil Company | Systems, methods and computer-readable media for monitoring a computer network for threats using OLAP cubes |
| CN118550672B (en) * | 2024-07-24 | 2024-10-22 | 广州越创智数信息科技有限公司 | Unified scheduling method and system for enterprise data warehouse tasks |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2480599A (en) * | 2010-05-17 | 2011-11-30 | Tech Universit T Muenchen | Hybrid OLTP and OLAP database |
| US9692785B2 (en) * | 2013-03-05 | 2017-06-27 | Pierce Global Threat Intelligence | Systems and methods for detecting and preventing cyber-threats |
| US9276959B2 (en) * | 2013-11-11 | 2016-03-01 | Amazon Technologies, Inc. | Client-configurable security options for data streams |
| CN104731939B (en) * | 2015-03-31 | 2018-05-01 | 浪潮集团有限公司 | Method for sharing data among tenants through data table |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
| CN109685519A (en) * | 2018-11-07 | 2019-04-26 | 国家电网有限公司客户服务中心 | A kind of method and system for use in carrying towards multi-tenant integration of services monitoring |
-
2019
- 2019-05-16 CN CN201910405548.7A patent/CN110119422B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110119422A (en) | 2019-08-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110119422B (en) | Data processing system and equipment for data warehouse of micro-credit tenant | |
| EP3120261B1 (en) | Dependency-aware transaction batching for data replication | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| KR101559206B1 (en) | Method of processing log data, and system operating the same | |
| US20190362004A1 (en) | Data platform fabric | |
| US20180113771A1 (en) | Transaction consistency query support for replicated data from recovery log to external data stores | |
| CN111897863B (en) | Multi-source heterogeneous data fusion and convergence method | |
| US11394794B2 (en) | Fast ingestion of records in a database using data locality and queuing | |
| TW202002587A (en) | Block chain-based data processing method and device | |
| US11263236B2 (en) | Real-time cross-system database replication for hybrid-cloud elastic scaling and high-performance data virtualization | |
| US9582558B2 (en) | Method and system for data definition language (DDL) replication | |
| US10095699B2 (en) | Computer-readable recording medium, execution control method, and information processing apparatus | |
| US10552293B2 (en) | Logging as a service | |
| US8738767B2 (en) | Mainframe management console monitoring | |
| US10776359B2 (en) | Abstractly implemented data analysis systems and methods therefor | |
| CN110188103A (en) | Data account checking method, device, equipment and storage medium | |
| CN113626447B (en) | Civil aviation data management platform and method | |
| CN107133231B (en) | Data acquisition method and device | |
| EP3769232B1 (en) | Method and system for performing inner-joins on a sharded data set | |
| CN106407442A (en) | Massive text data processing method and apparatus | |
| CN115640300A (en) | Big data management method, system, electronic equipment and storage medium | |
| Saranya et al. | Data migration using etl workflow | |
| CN106716400B (en) | A kind of partition management method and device of tables of data | |
| CN113626869A (en) | Data processing method, system, electronic device and storage medium | |
| US20210056120A1 (en) | In-stream data load in a replication environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |