CN102385628B - Java data base connectivity (JDBC)-based data distributed processing method - Google Patents

Abstract

The present invention relates to a kind of data distributed processing method based on JDBC, concrete steps are as follows: 1) install virtual database JDBC driver package file in client or Web application program, set up virtual database connection, and send SQL to virtual database according to JDBC standard Invoke request; 2) carry out parameter setting in virtual database, set up connection with each entity database by JDBC driver layer, with data cache and/or be stored in the multiple databases of distributed system; 3) the data distribution of described virtual database The processor receives the data SQL call request, analyzes the statement, and then distributes the execution to realize the distributed processing of the data. The present invention only needs to add a virtual database driver package to the class loading path of the original application program, and simply configure the virtual database to improve the data processing capability of the original program; the scalability is strong, and the data storage capacity can be improved through vertical expansion , and the concurrent processing capability can also be improved through horizontal expansion.

Description

A Distributed Data Processing Method Based on JDBC

technical field

The invention relates to data processing, in particular to a JDBC-based data distributed processing method, which belongs to the field of data storage and retrieval.

Background technique

When the scale of data processed by the application system continues to increase, there are generally two ways to meet the storage and processing requirements of these massive data. One is to use minicomputers and mainframes with higher performance, and the other is to use ordinary machines. of clusters. Because the cluster has an outstanding cost performance, it has largely replaced the mainframe, and the application on the Web server and application server is a good example. JDBC (Java Data Base Connectivity, java database connection) is a Java API for executing SQL statements, which can provide unified access to various relational databases. It consists of a set of classes and interfaces written in the Java language. JDBC provides a baseline against which higher-level tools and interfaces can be built to connect to a database, send statements that manipulate the database, and process the results. JVM is the abbreviation of (Java Virtual Machine, Java Virtual Machine). JVM is a specification for computing equipment. It is a fictitious computer that is realized by simulating various computer functions on an actual computer. The Java virtual machine includes a set of bytecode instructions, a set of registers, a stack, a garbage collection heap, and a storage method field. The JVM shields the information related to the specific operating system platform, so that the Java program only needs to generate the object code (byte code) that runs on the Java virtual machine, and it can run on various platforms without modification.

In data storage and processing, although clusters provide high performance and strong fault tolerance, there are far fewer tools in this area, and they are mainly solutions for large enterprises. Business solutions such as Oracle Real Application Clusters have begun to apply shared storage systems; IBM's DB2 Integrated Cluster environment uses network shared storage; and in terms of database clusters in the open source camp, MySQL replication technology is implemented using a master-slave mechanism. The obvious defect of these technologies is that in order to realize the characteristics of the cluster, the database engine must be extended, so the application program needs to use some additional APIs. API (Application Programming Interface, application programming interface) is some predefined functions, the purpose is to provide The ability for a developer to access a set of routines based on a piece of software or hardware without having access to the source code, or understanding the details of the inner workings. And it is difficult for different implementations of these different technologies to cooperate with each other.

Contents of the invention

The purpose of the present invention is to propose a data distributed processing method based on JDBC, which can convert a series of different types of entity databases into a unified virtual database. Load the JDBC driver of the virtual database, and send SQL requests to the virtual database according to the JDBC specification to realize data retrieval, insertion, update, and deletion operations on each entity database.

In order to solve the problems of the technologies described above, the technical solutions of the present invention include:

A JDBC-based data distributed processing method, the specific steps are as follows:

1) set the virtual database JDBC driver package file position in the class loading path of the client or web application developed based on the Java specification, so that the application program can load the driver package, set up a connection with the virtual database, and send the JDBC driver package according to the JDBC specification The virtual database sends a SQL call request;

2) The virtual database is a server-side application developed based on the Java specification. It uses configuration management tools to set parameters, so as to establish connections with each entity database through the drivers provided by the manufacturer in the JDBC driver layer, and cache and/or store data. in multiple databases in a distributed system;

3) The data distributed processor of the virtual database receives the data SQL call request, and analyzes the statement, and then distributes and executes, queries and/or updates the data in the entity database, and realizes the distributed processing of the data.

If the data volume of the application is large and exceeds the processing capacity of a single node, the data will be stored in a multi-node cluster; if the data volume of the application is small, the data will be backed up in multiple node clusters.

The data distributed processor includes: SQL call interface, SQL statement analyzer, SQL statement executor, scheduling distributor, data cache, connection pool, data access plug-in manager, Oracle data access, MySQL data access, user security, Modules such as log maintenance and configuration management.

The JDBC driver layer includes: an Oracle driver, a MYsql driver layer, and drivers provided by other manufacturers that meet the JDBC specification.

The databases include Oracle databases, MySQL databases and other vendor databases.

Described virtual database JDBC driving package: be a driver program conforming to JDBC specification, can run in JVM, make user and virtual database management system carry out communication, user's SQL sentence is sent in the virtual database, and its result will be sent back to the user.

The present invention is mainly used in a specific environment, if the amount of applied data is large and exceeds the processing capacity of a single node, this method can be used to store data in a multi-node cluster in a vertical expansion manner, and meet the requirements through distributed processing ; If the amount of data in the application is small, but the user has many concurrent requests, this method can be used in a horizontal expansion mode to back up the data multiple times in multiple node clusters, and improve the concurrent response capability by load balancing the requests.

Beneficial effects of the present invention:

1. The application is simple. You only need to add a virtual database driver package to the class loading path of the original application program, and simply configure the virtual database to improve the data processing capability of the original program and adapt to data processing of different data volumes.

2. Strong scalability, the data storage capacity can be improved through vertical expansion, and the concurrent processing capacity can also be improved through horizontal expansion.

Description of drawings

Fig. 1 is the implementation schematic diagram of the data distributed processing method described in the present invention;

Fig. 2 is a flow chart of the data distributed processing method according to the present invention;

Detailed ways

The preferred embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the virtual database JDBC driver package file position is set in the class loading path of the client or Web application developed based on the Java specification, so that the application program can load the driver package and establish a connection with the virtual database, according to The JDBC specification sends SQL search, insert, update, delete and other operation call requests to the virtual database; the virtual database is a server-side application developed based on the Java specification, and uses configuration management tools to set parameters so that it can be provided by the manufacturer in the JDBC driver layer. The driver establishes connections with each entity database respectively, and caches and/or stores the data in multiple databases of the distributed system; after the virtual database receives the SQL call request from the application program, it analyzes the statement through the data distributed processor, and then Distribute execution to realize distributed processing of data. Data distributed processor includes SQL call interface, SQL statement analyzer, SQL statement executor, scheduling distributor, data cache, connection pool, data access plug-in manager, Oracle data access , MySQL data access, user security, log maintenance, configuration management and other modules.

The entity database can be an Oracle database, a MySQL database, a SQL Server database, or a database from other manufacturers, as long as the corresponding data access plug-in is developed, and then deployed in the data distributed processor.

As shown in Figure 2, the present invention proposes a kind of data distributed processing method based on JDBC, taking Web application program as example, its processing flow is as follows:

(1) In the web application program, according to the JDBC specification, encapsulate the SQL search, insert, update and delete operation call requests;

(2) Through the JDBC driver package of the virtual database, the Web application program establishes a connection with the virtual database, and sends the SQL calling request encapsulated in step 1) to the virtual database through this connection.

(3) The virtual database manages all connections established with the Web application through the connection pool module, and authorizes and audits the SQL operations of the Web application through the user security and log maintenance module.

(4) For the received SQL call request, convert it into a SQL call statement supported by the virtual database through the SQL call interface.

(5) According to the parameter configuration of the virtual database, the SQL statement analyzer determines which tables are involved in this SQL call request, which data access plug-ins are required to participate, and the processing method of the result set, and assembles them into execution commands that can be recognized by each data access plug-in , and then send the execution command to the SQL statement executor. According to the specific situation of the business application, the parameter configuration of the virtual database is different: for a table with a large amount of data, the parameter configuration can store the data in multiple actual databases; for a table with a small amount of data, redundant storage can be performed, and then through Multiple backups to increase system reliability and concurrent performance. The SQL statement analyzer can set the result set processing method according to different business application situations. For the decentralized storage mode, the multi-database result set merge processing method is adopted; for the multiple backup mode, load balancing is adopted to select the best database to execute and return the result.

(6) After the SQL statement executor receives the execution command, it first judges whether the cache can be used and the cached data is valid. If so, it will directly call the data cache module to return the result to the Web application, otherwise it will send the execution command to the dispatcher for execution. According to the result set processing method set by the execution command, process the result set returned by the dispatcher, then return the result to the web application, and call the data cache module to update the cache.

(7) The scheduling dispatcher obtains the corresponding data access plug-in from the data access plug-in manager according to the command type for the received execution command, then calls the corresponding interface of the data access plug-in to execute the command, and returns the result to the SQL statement executor for processing . If multiple entity databases are involved, each will be scheduled for execution.

(8) The data access plug-in encapsulates the access interface of the entity database, generally establishes a connection with the entity database through a JDBC driver, sends the SQL call request to the entity database management system for execution and collects the returned results. Databases from different vendors have their own data access plug-ins, such as Oracle data access plug-ins, MySQL data access plug-ins, SQL Server data access plug-ins, etc. All data access plug-ins need to be registered in the data access plug-in manager through the virtual database configuration management tool .

(9) The user security, log maintenance, and configuration management modules provided by the data distributed processor are basic modules that provide configuration, authorization, and audit services for other modules to increase system security and reliability.

Claims (5)

1., based on a Data distribution8 formula disposal route of JDBC, its step comprises:

1) virtual data base JDBC is set in client or web application and drives APMB package, set up virtual data base and connect, and send SQL call request according to JDBC specification to virtual data base;

2) virtual data base carries out optimum configurations, drives the driver in layer and each entity data bak to connect by JDBC, by data buffer storage and/or be stored in multiple databases of distributed system; According to the concrete condition of service application, the parameter configuration of virtual data base is different: for the table that there is mass data, data scatter is stored in the database of multiple reality; For the table of small data quantity, carry out redundant storage, and then increase reliability and the concurrency performance of system by multiple duplication;

3) the Data distribution8 formula processor of described virtual data base receives data SQL call request, and line statement analysis of going forward side by side, then distributes execution, realizes the distributed treatment of data; Described Data distribution8 formula processor comprises:

SQL calling interface, converts the SQL call statement that virtual data base is supported to by the SQL call request received;

SQL statement analyzer, carries out Sentence analysis to the SQL call request received, and arranges result set processing mode according to different service application situations: for dispersion memory module, adopt many storehouses result set merger processing mode; For multiple duplication pattern, adopt load balancing, select best database perform and return results;

SQL statement actuator, receives sql command from SQL statement analyzer and performs;

Scheduling distributor, the SQL realizing multinode entity data bak calls distribution and performs, from Data access plug-ins manager, corresponding Data access plug-ins is obtained according to the fill order classification received, then calling data access plug-in unit the corresponding interface fill order, and result is returned to SQL statement actuator and process;

Data buffer, for the Data Update of buffer memory JDBC.

2. disposal route as claimed in claim 1, is characterized in that, described JDBC drives layer to comprise: Oracle driving, MySQL driving, SQL Server driving and other manufacturers meet the driving of JDBC specification.

3. disposal route as claimed in claim 1, it is characterized in that, described entity data bak comprises oracle database, MySQL database, SQL Server database.

4. disposal route as claimed in claim 1, it is characterized in that, described Data distribution8 formula processor also comprises Data access plug-ins manager.

5. disposal route as claimed in claim 1, it is characterized in that, described Data distribution8 formula processor also comprises user security, daily record maintenance and Configuration Manager.