CN113590594B

CN113590594B - Bank database migration method and device

Info

Publication number: CN113590594B
Application number: CN202110980892.6A
Authority: CN
Inventors: 朱晓艳; 廖文胜; 卢盛伟; 蔡万清
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2024-10-15
Anticipated expiration: 2041-08-25
Also published as: CN113590594A

Abstract

The invention discloses a bank database migration method and a device, and relates to the technical field of big data, wherein the method comprises the following steps: generating a plurality of data intermediate files according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located; determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated; screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file, and generating a target file; and loading the target file into a target database. The invention can improve the migration efficiency of the bank database, reduce the consumption of memory, IO and CPU resources of the bank business system and lighten the data processing pressure of the bank database and the bank business system.

Description

Bank database migration method and device

Technical Field

The invention relates to the technical field of big data, in particular to a bank database migration method and device.

Background

This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

The customer information field is one of the most basic fields of banks, is the basis for customer service, meeting supervision requirements and internal management of banks, and is characterized in that in the comprehensive treatment work of customer information, customers are required to be classified according to factors such as the integrity degree of customer information elements, the activity degree of customers and the like, and customers with lower customer information quality levels are isolated and managed, only incomplete basic financial service is provided, and the customers are required to be taken as dimensions, and the information such as account, card, business relation and the like under the customer names is stripped from an original system and accepted by a newly-built system.

The data in the part of the source database table is screened, extracted, converted and loaded into the target system according to a certain rule, and the process is called data migration.

For large-scale application systems, with the continuous expansion of service range and long-term operation of the system, the accumulation of service data is very huge, association relation is complex, and in addition, in order to improve access and storage efficiency to a certain extent, some table structures adopt a wide table design, which makes data migration work have great challenges in quantity, complexity and diversity.

At present, when the bank database is migrated, the direct SELECT operation can be adopted to perform the SELECT operation on the bank data table, and the huge amount of business data in the bank database can cause a great amount of consumption of memory, IO and CPU resources of the bank business system and also can cause the reduction of the migration efficiency of the bank database.

Disclosure of Invention

The embodiment of the invention provides a bank database migration method, which relates to the technical field of big data, and is used for improving the migration efficiency of a bank database, reducing the consumption of memory, IO and CPU resources of a banking system and relieving the data processing pressure of the bank database and the banking system, and the method comprises the following steps:

Generating a plurality of data intermediate files according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located;

determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated;

Screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file, and generating a target file;

And loading the target file into a target database.

The embodiment of the invention also provides a bank database migration device, which relates to the technical field of big data, and is used for improving the migration efficiency of the bank database, reducing the consumption of memory, IO and CPU resources of a banking system and relieving the data processing pressure of the bank database and the banking system, and the device comprises:

the data intermediate file generation module is used for generating a plurality of data intermediate files according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located;

The bank account card number to be migrated determining module is used for determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated;

The target file generation module is used for screening out data to be migrated, which are associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated, from each data intermediate file to generate a target file;

and the target file loading module is used for loading the target file into the target database.

The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the bank database migration method when executing the computer program.

The embodiment of the invention also provides a computer readable storage medium, which stores a computer program for executing the bank database migration method.

In the embodiment of the invention, a plurality of data intermediate files are generated according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located; determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated; screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file, and generating a target file; the target file is loaded to the target database, so that the migration work of the data table in the bank database can be converted into the processing of the data intermediate file by generating the data intermediate file corresponding to each disk partition, the problems that the internal memory, IO and CPU resources of the bank system are greatly consumed when the data table is directly migrated and the data processing pressure of the bank database and the bank system is increased due to the fact that the record length and the data volume of the data table are large in the prior art are solved, the internal memory, IO and CPU resources of the bank system can be effectively reduced by generating the data intermediate file, and the data processing pressure of the bank database and the bank system is reduced; meanwhile, to-be-migrated data corresponding to each client number is selected through screening, so that the migration range of the bank database can be effectively defined, data needing to be migrated can be accurately obtained, and the migration efficiency of the bank database is improved.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:

FIG. 1 is a schematic flow chart of a method for migrating a bank database according to an embodiment of the present invention;

FIG. 2 is a diagram showing a bank database migration method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a bank database migration apparatus according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a bank database migration apparatus according to an embodiment of the present invention;

FIG. 5 is a diagram illustrating a bank database migration apparatus according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating an example of generating a data intermediate file in a bank database migration apparatus according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a specific example of a data intermediate file in a bank database migration apparatus according to an embodiment of the present invention;

FIG. 8 is a diagram illustrating a bank database migration apparatus according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating a bank database migration apparatus according to an embodiment of the present invention;

Fig. 10 is a schematic diagram of a computer device for bank database migration in an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present invention and their descriptions herein are for the purpose of explaining the present invention, but are not to be construed as limiting the invention.

Embodiments of the present invention relate to the following terms, as explained below:

Mainframe: or a large host, english name mainframe. The mainframe uses a dedicated processor instruction set, operating system and application software. The term mainframe is initially used to refer to a mainframe computer system that is housed in a very large, framed iron box, to distinguish it from smaller mini-and micro-machines.

And (3) data migration: the method is a data conversion process for transiting various business data from a database table of an old system to a corresponding database table of a new system in order to protect the business coverage of an original system and the application function of the original system under the scenes of replacing the old system and the new system, transferring different business varieties in the old system and the like. Data migration can be classified into isomorphic system data migration and heterogeneous system data migration according to the system environment.

Broad table: i.e. database tables with a relatively large number of fields.

UNLOAD: data is offloaded from one or more database targets into one or more QSAM sequential data sets.

At present, the customer information field is one of the basic fields of banks, is the basis for customer service, meeting the supervision requirements and internal management of the banks, and is characterized in that in the comprehensive treatment work of customer information, customers are required to be classified according to factors such as the integrity degree of customer information elements, the activity degree of the customers and the like, and customers with lower customer information quality levels are isolated and managed, only incomplete basic financial services are provided, and the customers are required to be taken as dimensions, and the information such as account, card, business relation and the like under the customer names is stripped from the original system and accepted by a newly-built system. The data in the part of the source database table is screened, extracted, converted and loaded into the target system according to a certain rule, and the process is called data migration.

Direct SELECT operation on the data table requires a large amount of IO and CPU overhead; the screening efficiency can be greatly improved by orderly recording the files, but under the conditions of long recording and large data volume, the files are directly sequenced, and a large amount of IO, memory and CPU resources are consumed, so that the whole migration window is influenced.

In order to solve the above problems, an embodiment of the present invention provides a method for migrating a bank database, which can be used to improve the migration efficiency of bulk data of a bank, reduce the consumption of memory, IO and CPU resources of a banking system in the migration process, and reduce the data processing pressure of the bank database and the banking system, as shown in fig. 1, and the method may include:

Step 101: generating a plurality of data intermediate files according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located;

step 102: determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated;

Step 103: screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file to generate a target file;

step 104: and loading the target file into a target database.

In the embodiment of the invention, a plurality of data intermediate files are generated according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located; determining a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated; screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file to generate a target file; the target file is loaded to the target database, so that the migration work of a data table in the bank database can be converted into the processing of the data intermediate file by generating the data intermediate file corresponding to each disk partition, and the sorting of the wide files is converted into the sorting of the narrow files, thereby solving the problems that the prior art can cause great consumption of memory, IO and CPU resources of the bank service system when the data table is directly migrated, increasing the data processing pressure of the bank database and the bank service system, and effectively reducing the consumption of the memory, IO and CPU resources of the bank service system and relieving the data processing pressure of the bank database and the bank service system by generating the data intermediate file; meanwhile, to-be-migrated data corresponding to each client number is selected through screening, so that the migration range of the bank database can be effectively defined, data needing to be migrated can be accurately obtained, and the migration efficiency of the bank database is improved.

When the method is implemented, firstly, a plurality of data intermediate files are generated according to a data table to be migrated in a bank database; the data table to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data table to be migrated is located.

In an embodiment, the data intermediate file may be QSAM (queued sequential access, in-line sequential access) data file.

In the above embodiment, the data intermediate file corresponding to each disk partition can be generated, so that the wide record file can be converted into the narrow record file under the conditions of multiple table fields, huge data volume and complex table association relationship, the migration work of the data table in the bank database can be converted into the processing of the data intermediate file, the problems that the data intermediate file is generated, the data intermediate file can cause great consumption of memory, IO and CPU resources of the bank database and the bank business system when the data table is directly migrated in the prior art are solved, the data processing pressure of the bank database and the bank business system is increased, and the consumption of the memory, IO and CPU resources of the bank business system caused by the data migration can be effectively reduced, so that the data processing pressure of the bank database and the bank business system is lightened.

In one embodiment, generating a plurality of data intermediate files according to the data table to be migrated in the bank database may include: and generating a plurality of data intermediate files according to the data table to be migrated in the bank database by adopting a database UNLOAD technology.

In the above embodiment, by exporting the table data to be migrated into QSAM data files, the operation on the physical table can be converted into the processing of file records, so that the CPU and IO consumption is reduced.

For example, the above-mentioned data table to be migrated in the bank database generates a plurality of data intermediate files, and may adopt the UNLOAD technology to store the table record to be migrated as the data intermediate file, and convert the lookup operation of the database table into the operation of the file; meanwhile, the characteristics of the partition table can be fully utilized, and each partition can generate a data file according to the concurrent processing of the partitions.

In this example, the table data UNLOAD is stored in QSAM files, and the file recording order is consistent with the disk storage order.

In this example, the specific step of generating a plurality of data intermediate files according to the data table to be migrated in the bank database may be implemented by the codes in table 1.

TABLE 1

The pseudocode of the code in Table 1 is as follows:

in the specific implementation, after a plurality of data intermediate files are generated according to a data table to be migrated in a bank database, a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated, are determined; and screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file to generate a target file.

In an embodiment, screening to-be-migrated data associated with a to-be-migrated bank account number and a to-be-migrated bank card number corresponding to a to-be-migrated client number from each data intermediate file, and generating the target file may include:

matching each client number in the data intermediate file with the client number to be migrated according to each data intermediate file, and screening to obtain data to be migrated corresponding to the client number to be migrated in the data intermediate file;

Generating a target file according to the data to be migrated corresponding to the client number to be migrated in each data intermediate file;

And/or the number of the groups of groups,

Matching a bank account corresponding to each client number in each data intermediate file with a bank account to be migrated, and screening to obtain data to be migrated, corresponding to the bank account to be migrated, in the data intermediate file;

Generating a target file according to the data to be migrated corresponding to the bank account number to be migrated in each data intermediate file;

And/or the number of the groups of groups,

Matching the bank card number corresponding to each client number in each data intermediate file with the bank card number to be migrated, and screening to obtain data to be migrated, wherein the data corresponds to the bank card number to be migrated in the data intermediate file;

and generating a target file according to the data to be migrated corresponding to the bank card number to be migrated in each data intermediate file.

In one embodiment, the data to be migrated associated with each customer number to be migrated, the bank account number to be migrated, and the bank card number to be migrated is screened from each data intermediate file, and the generation of the target file may include, as shown in fig. 2:

Step 201: screening data to be migrated, which are associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated, from each data intermediate file;

Step 202: and screening the data to be migrated from each data intermediate file, and generating a target file according to a preset file format.

In the above embodiment, the target file is generated by screening the to-be-migrated data associated with the to-be-migrated bank account number and the to-be-migrated bank card number corresponding to the to-be-migrated customer number from each data intermediate file, so that the to-be-migrated data corresponding to each customer number can be screened out from each data intermediate file at the same time, the purpose of concurrently processing the migration data of the bank database according to the partition can be realized, the migration efficiency of the bank database is improved, and the concurrency of the migration operation of the bank database is improved.

In the above embodiment, the migration table may be configured to output 1 to N data intermediate files (the specific number mainly considers the processing efficiency, the storage space and the concurrency upper limit) according to the partition UNLOAD, fully comb the association relationship between the migration tables, determine the screening factor and the sorting factor of each data table to be migrated, simplify the "wide record" file into the "narrow record" file, fully sort the files, and screen the migrated data of the client-related table, the account-related table and the card-related table respectively on the basis of the ordered files, and finally form the target migration file.

In the specific implementation, a bank account number to be migrated and a bank card number to be migrated, which correspond to each customer number to be migrated, are determined; and screening data to be migrated, which are associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated, from each data intermediate file, and loading the target file into a target database after generating the target file.

In the embodiment, the migration of the target data to be migrated is realized by loading the target file into the target database. In an embodiment, by matching the bank account number corresponding to each client number in the data intermediate file with the bank account number to be migrated according to the above data intermediate file, screening to obtain data to be migrated corresponding to the bank account number to be migrated in the data intermediate file, and matching the bank card number corresponding to each client number in the data intermediate file with the bank card number to be migrated according to each data intermediate file, screening to obtain data to be migrated corresponding to the bank card number to be migrated in the data intermediate file, the screening factor (i.e. the matching item when performing table record screening, where the "client number to be migrated" as above is a screening factor) can be indirectly determined.

In specific implementation, the method for migrating a bank database provided by the embodiment of the invention can further include: generating a data table list to be migrated according to the business information of the data migration of the bank database;

determining the disk partition information of each data table to be migrated in the data table list to be migrated;

and obtaining the data table to be migrated in the bank database according to the disk partition information of each data table to be migrated in the data table list to be migrated.

In one embodiment, the service information may include service rules preset by the staff and design rules of the data migration system, and may be freely set according to the use requirement of the staff.

In an embodiment, the list of the data table to be migrated can be determined by combing according to the design and service functions of the banking system, and the method is helpful for collecting the whole data of the migration table by using the UNLOAD technology in the previous step, so as to generate QSAM data files.

In the above embodiment, the data to be migrated in the bank database may be finally obtained by generating the list of the data tables to be migrated, determining the partition information of the disk where each data table to be migrated is located in the list of the data tables to be migrated. Because of the high concurrent large-scale batch data migration process in the bank database at present, the number of executed migration operations and the generated intermediate files are huge, the total record number of the migration files of each migration table and the final migration record number of the target database can be counted through the steps, and the migration accuracy and the migration integrity are ensured.

In specific implementation, the method for migrating a bank database provided by the embodiment of the invention can further include:

And pre-configuring the number of the client to be migrated according to the hierarchical classification information of the client.

In the embodiment, according to the relation between the customer number and the account number and the card number, corresponding lists such as the bank account to be migrated, the bank card number to be migrated and the like can be generated, and the migration range is defined.

In the embodiment, the client number to be migrated is preconfigured according to the hierarchical classification information of the clients, so that the data to be migrated of the bank databases in different batches can be accurately determined.

In the implementation, for each data intermediate file, matching the bank account corresponding to each client number in the data intermediate file with the bank account to be migrated, and screening to obtain data to be migrated corresponding to the bank account to be migrated in the data intermediate file, which may include:

for each data intermediate file:

extracting all bank accounts from the data intermediate file;

adding a number to each bank account to obtain a first file; the natural number sequence numbered from 1 can be used for identifying the position of the strip recorded in the data intermediate file; sequencing the first files according to a preset sequence of bank accounts to obtain second files; the preset sequence comprises ascending sequence or descending sequence;

matching the bank account to be migrated (the bank account to be migrated is the sequenced bank account to be migrated) with the bank account in the second file, and screening the bank account in the second file successfully matched and the number corresponding to the bank account;

generating a third file according to the screened bank account number in the second file successfully matched and the serial number corresponding to the bank account number;

Arranging the bank accounts in the third file according to a preset sequence (such as ascending sequence) of numbers to obtain a fourth file;

Matching the serial numbers of the bank accounts in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file after the serial numbers to obtain the data to be migrated corresponding to the bank accounts to be migrated in the data intermediate file; the numbered data intermediate file is obtained by adding the number to the data intermediate file.

The record number of the data intermediate file is obtained by counting the file sequence traversal process.

In one embodiment, for each data intermediate file, matching the bank card number corresponding to each client number in the data intermediate file with the bank card number to be migrated, and screening to obtain data to be migrated corresponding to the bank card number to be migrated in the data intermediate file may include:

for each data intermediate file:

Extracting all bank card numbers from the data intermediate file;

Adding a number to each bank card number to obtain a first file; the natural number sequence numbered from 1 can be used for identifying the position of the strip recorded in the data intermediate file; sequencing the first files according to a preset sequence of bank card numbers to obtain second files; the preset sequence comprises ascending sequence or descending sequence;

Matching the bank card number to be migrated (namely the ordered bank card number to be migrated) with the bank card number in the second file, and screening the bank card number in the second file successfully matched with the bank account number;

generating a third file according to the screened bank card number in the second file successfully matched with the bank account number and the serial number corresponding to the bank account number;

Arranging the bank card numbers in the third file according to a preset sequence (such as ascending sequence) of the numbers to obtain a fourth file;

Matching the serial number of the bank card number in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file to obtain the data to be migrated corresponding to the bank card number to be migrated in the data intermediate file; the numbered data intermediate file is obtained by adding the number to the data intermediate file.

In an embodiment, the "wide record" file to be screened is reduced to a "narrow record" file containing only the screening factor data item, and a "sequence number" column is added to the "narrow record" file to mark the location of the record in the file. Meanwhile, the process can also adopt DFSORT sorting tools to sort the files of the narrow records for multiple times, ensure that each screening and matching operation is carried out on the basis of ordered files, greatly save IO, memory and CPU consumption of the system and shorten the overall data migration time.

In the above embodiment, due to the high concurrent large-scale batch data migration process in the current bank database, the number of migration operations executed and the generated intermediate files are very huge, and the total record number of migration files of each migration table and the final migration record number of the target database can be counted through the steps, so that the migration accuracy and integrity are ensured.

Generating a list of data tables to be migrated according to the business information of the data migration of the bank database;

loading the target file into the target database may include:

and loading the target file into the data table to be migrated in the target database according to the disk partition information of each data table to be migrated in the data table list to be migrated.

In an embodiment, according to the disk partition information of each to-be-migrated data table in the to-be-migrated data table list, the target file is loaded to the to-be-migrated data table in the target database, so that accurate transfer of the to-be-migrated data from the migration database to the migration database can be realized.

And recording to-be-migrated data and generated target files, wherein the to-be-migrated data are associated with the to-be-migrated bank account number and the to-be-migrated bank card number, and the to-be-migrated data correspond to the to-be-migrated customer number, and the to-be-migrated data are screened from each data intermediate file.

In the embodiment, by adding the migration intermediate file packaging and storing function, the traceability and rollback of the migration process are ensured.

In the embodiment of the invention, a plurality of data intermediate files are generated according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located; screening out to-be-migrated data associated with a to-be-migrated bank account number and a to-be-migrated bank card number corresponding to a to-be-migrated client number from each data intermediate file, and generating a target file; the target file is loaded to the target database, so that the migration work of the data table in the bank database can be converted into the processing of the data intermediate file by generating the data intermediate file corresponding to each disk partition, the problems that the internal memory, IO and CPU resources of the bank system are greatly consumed when the data table is directly migrated and the data processing pressure of the bank database and the bank system is increased due to the fact that the record length and the data volume of the data table are large in the prior art are solved, the internal memory, IO and CPU resources of the bank system can be effectively reduced by generating the data intermediate file, and the data processing pressure of the bank database and the bank system is reduced; meanwhile, to-be-migrated data corresponding to each client number is selected through screening, so that the migration range of the bank database can be effectively defined, data needing to be migrated can be accurately obtained, and the migration efficiency of the bank database is improved.

As described above, the embodiment of the present invention provides a data migration method for a large-scale host based application system, which is capable of implementing a universal and efficient batch bank database, and under the conditions of multiple table fields, huge data volume and complex table association relationship, converting a "wide record" file into a "narrow record" file, and combining DFSORT technologies, performing data screening on the basis of basically ordered files, and simultaneously performing segmentation parallel processing by using the table partition characteristics, so as to improve the operation concurrency and execution efficiency, and on the premise of ensuring normal operation of the original system for 7×24 hours, rapidly and accurately screening, converting migration data from a source database, and loading the migration data into a target system.

The embodiment of the invention can convert the database table UNLOAD into a data file, convert the table operation into file processing, and reduce the system resource consumption; the wide record file can be skillfully simplified into the narrow record file, so that the ordering efficiency is greatly improved; meanwhile, the principle of table partitioning can be utilized, and the processing is performed according to partitioning concurrency, so that the processing efficiency is greatly improved.

The embodiment of the invention also provides a bank database migration device, and the embodiment is as follows. The principle of the device for solving the problems is similar to that of the bank database migration method, so that the implementation of the device can refer to the implementation of the bank database migration method, and the repeated parts are not repeated.

The bank database migration device provided by the embodiment of the invention can be used for improving the migration efficiency of a bank database, reducing the consumption of memory, IO and CPU resources of a banking system and relieving the data processing pressure of the bank database and the banking system, and as shown in fig. 3, the device can comprise:

The data intermediate file generation module 01 is used for generating a plurality of data intermediate files according to the data table to be migrated in the bank database; the data to be migrated is stored in a plurality of disk partitions of the host, and a plurality of data intermediate files respectively correspond to different disk partitions of the host where the data to be migrated is located;

The to-be-migrated bank account card number determining module 02 is configured to determine a to-be-migrated bank account number and a to-be-migrated bank card number corresponding to each to-be-migrated customer number;

the target file generating module 03 is configured to screen data to be migrated associated with each customer number to be migrated, the bank account number to be migrated, and the bank card number to be migrated from each data intermediate file, and generate a target file;

And the target file loading module 04 is used for loading the target file into the target database.

In one embodiment, as shown in fig. 4, a bank database migration apparatus provided in the embodiment of the present invention may further include:

the data table to be migrated determination module 05 is configured to:

generating a data table list to be migrated according to the business information of the data migration of the bank database;

In one embodiment, the bank database migration apparatus provided by the embodiment of the present invention may further include:

the client number configuration module to be migrated is used for:

In one embodiment, the object file generation module is specifically configured to:

And/or the number of the groups of groups,

for each data intermediate file:

extracting all bank accounts from the data intermediate file;

adding a number to each bank account to obtain a first file; the above numbers are natural number sequences starting from 1; the serial numbers in the first file are sequenced according to the preset sequence of the bank account numbers, and a second file is obtained; the preset sequence comprises ascending sequence or descending sequence;

Matching the bank account to be migrated with the bank account in the second file, and screening the bank account in the second file successfully matched and the serial number corresponding to the bank account;

Arranging the bank accounts in the third file according to the preset sequence of the numbers to obtain a fourth file;

for each data intermediate file:

Extracting all bank card numbers from the data intermediate file;

adding a number to each bank card number to obtain a first file; the above numbers are natural number sequences starting from 1; the serial numbers in the first file are sequenced according to the preset sequence of the bank card numbers, and a second file is obtained; the preset sequence comprises ascending sequence or descending sequence;

matching the bank card number to be migrated with the bank card number in the second file, and screening the bank card number in the second file successfully matched and the number corresponding to the bank account number;

arranging the bank card numbers in the third file according to the preset sequence of the numbers to obtain a fourth file;

Matching the serial number of the bank card number in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file after the serial number to obtain the data to be migrated corresponding to the bank card number to be migrated in the data intermediate file; the numbered data intermediate file is obtained by adding the number to the data intermediate file.

Screening data to be migrated, which are associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated, from each data intermediate file;

And screening the data to be migrated from each data intermediate file, and generating a target file according to a preset file format.

the disk partition information determining module is used for:

the target file loading module is specifically configured to:

A specific embodiment is given below to illustrate a specific application of the device of the present invention, in which embodiment, as in fig. 5, may include the following:

(1) A data table UNLOAD module (i.e. the data intermediate file generating module and the data table to be migrated determining module) described above: according to the system design and the service function, a data list to be migrated is determined by carding, and full data of the data list to be migrated is collected by using an UNLOAD technology to generate QSAM data files;

(2) A screening factor determining module (namely the client number configuration module to be migrated and the target file generating module) which are described above: determining an unnormalized client number list according to migration rules, and generating corresponding migration account numbers, card numbers and other lists according to the relation between the client numbers, the account numbers and the card numbers, namely defining migration ranges for subsequent screening of related table records;

(3) Data screening module (i.e. the target file generation module described above): screening out data meeting migration conditions in a migration table;

(4) Target format conversion module (i.e., target file loading module described above): integrating the migration data into a target format to generate a final migration file;

(5) Target system loading module (i.e., target file loading module described above): loading migration file data to a target system database;

(6) Source system data deletion module: logically deleting or physically deleting the migrated data of the source system database;

(7) Migration result checking module (i.e. the above-mentioned object file generating module): it is checked whether the migration data and the migration data agree.

The apparatus in this example is described in detail below with reference to the drawings:

1. the data table UNLOAD module is specifically configured to:

determining a data table list to be migrated according to the business information of the data migration of the bank database;

Obtaining the data tables to be migrated in the bank database according to the disk partition information of each data table to be migrated in the data table list to be migrated;

And generating a plurality of data intermediate files according to the data table to be migrated in the bank database.

In this module, as shown in fig. 6, a plurality of data intermediate files can be generated according to the data table to be migrated in the bank database, so that the range of the migration table can be determined, and the table record requiring data migration is stored as a data file by adopting the UNLOAD technology, so that the search operation of the database table is converted into the operation of the file; meanwhile, the characteristics of the partition table are fully utilized, and each partition generates a data file according to the concurrent processing of the partitions.

As shown in fig. 7, the table data UNLOAD is stored as QSAM files, and the file recording order is consistent with the disk storage order. As shown in table 2, fig. 7 is explained, and one line of records in the file corresponds to one complete record in the table.

TABLE 2

Column	Type & length	Meaning of
			Organization number	char(3)	“003”
Partition number	char(3)	Partition number of table
			Card number	char(19)	Card number
Product numbering	char(10)	Card product number
			BIN numbering	char(5)	Card BIN number
Customer number	char(16)	Customer numbering
			……	……	……

2. Screening factor determination module

(1) Classifying the tables, such as a client class table, an account class table and a card class table, namely the client number configuration module to be migrated, which can be used for pre-configuring the client number to be migrated according to the hierarchical classification information of the client;

(2) Classifying the quality of the client information according to a personal and enterprise client information management method, and determining an nonstandard client number list to be isolated; it should be noted that, in this example, the client number to be migrated may be an irregular client number list to be isolated;

(3) And screening and generating an irregular account list and a card number list through the client and account relation table and the client and card relation table, and sorting according to ascending order to be used as screening factors of subsequent various tables.

In the whole process, the partition characteristics of the database table can be fully utilized, each partition generates an UNLOAD file, and a plurality of partitions are processed in parallel, so that the processing time is effectively reduced.

The workflow of the above-described screening factor determination module may be as shown in fig. 8. The screening factors correspond to customer numbers to be migrated, and different screening factors, such as customer numbers to be migrated, bank account numbers to be migrated, bank card numbers to be migrated and the like, can be set in different batches of data migration tasks, and can be flexibly set according to the use requirements of staff.

3. The data screening module, namely the target file generating module, is specifically configured to: for each data intermediate file: extracting all bank accounts from the data intermediate file; adding a number to each bank account to obtain a first file; the above numbers are natural number sequences starting from 1; the serial numbers in the first file are sequenced according to the preset sequence of the bank account numbers, and a second file is obtained; the preset sequence comprises ascending sequence or descending sequence; matching the bank account to be migrated with the bank account in the second file, and screening the bank account in the second file successfully matched;

Generating a third file according to the screened bank account numbers in the second file successfully matched; arranging the bank accounts in the third file according to the preset sequence of the numbers to obtain a fourth file; matching the serial numbers of the bank accounts in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file after the serial numbers to obtain the data to be migrated corresponding to the bank accounts to be migrated in the data intermediate file; the numbered data intermediate file is obtained by adding the number to the data intermediate file.

And a target file generation module, specifically configured to: for each data intermediate file: extracting all bank card numbers from the data intermediate file; adding a number to each bank card number to obtain a first file; the above numbers are natural number sequences starting from 1; sequencing the first files according to a preset sequence of bank card numbers to obtain second files; the preset sequence comprises ascending sequence or descending sequence; matching the bank card number to be migrated with the bank card number in the second file, and screening the bank card number in the second file successfully matched; generating a third file according to the screened bank card number in the second file successfully matched; arranging the bank card numbers in the third file according to the preset sequence of the numbers to obtain a fourth file; matching the serial numbers of the bank card numbers in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file according to the sequence to obtain the data to be migrated corresponding to the bank card numbers to be migrated in the data intermediate file; the record number of the data intermediate file is obtained by counting the file sequence traversal process.

The data screening module is the most important part in the whole migration process, and takes the deposit class table processing process as an example for explanation, as shown in fig. 9, all records meeting the screening conditions in the account expansion table INVE can be screened out according to the non-standard account list:

the workflow of the data screening module is specifically described according to the following steps:

(1) Traversing the full UNLOAD data file of INVE table, extracting an account column, adding a sequence number column, starting from 1, numbering and sequentially increasing the sequence number, marking the position of the bar recorded in the UNLOAD file, and outputting a file F1 with a structure of sequence number plus account;

(2) Ordering the files F1 according to the account numbers, wherein the order is consistent with the order of the files of the accounts with different specifications, and generating files F2 ordered according to the account numbers;

(3) Traversing the non-standard account list and the file F2 sequentially, and screening out non-standard account records in the F2 according to account matching to generate an intermediate screening file F3 with a structure of a sequence number plus account;

(4) The files F3 are sequenced according to the ascending sequence of the sequence number, files F4 ordered according to the sequence number are generated, and the recording sequence of the files F4 is consistent with that of the full UNLOAD files;

(5) And traversing the file F4 and the full-quantity UNLOAD file sequentially, matching according to sequence numbers, screening out complete records corresponding to the non-standard account number in the INVE table, and forming a final screening result, namely a INVE table migration file F5.

In the whole screening process, the partition characteristics of the database table can be fully utilized, each partition generates an UNLOAD file, and a plurality of partitions execute the following processes in parallel, so that the processing time is effectively reduced.

4. A target format conversion module, configured to: and screening out to-be-migrated data associated with the to-be-migrated bank account number and the to-be-migrated bank card number corresponding to the to-be-migrated client number from each data intermediate file, and generating a target file.

And processing the data to be migrated according to a certain mapping logic to generate a target file. May be performed concurrently by the table partition property.

5. Target system loading module

And loading the target file into a target database according to certain mapping logic. May be performed concurrently by the table partition property.

6. Source system data deleting module

And (3) on the basis of ensuring that the data is correctly loaded in the target system, logically deleting or physically deleting the migrated data of the source system according to the requirements according to the migration file generated in the step (3). May be performed concurrently by the table partition property.

7. Migration result checking module

And checking the total migration record number in each table migration file and the migration record number of the target database through statistics, so as to ensure the correctness of the migration result.

Of course, it is to be understood that other variations of the above detailed procedures are also possible, and all related variations should fall within the protection scope of the present invention.

The embodiment of the invention provides a computer device for realizing all or part of contents in the bank database migration method, which specifically comprises the following contents:

A processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between related devices; the computer device may be a desktop computer, a tablet computer, a mobile terminal, or the like, and the embodiment is not limited thereto. In this embodiment, the computer device may be implemented with reference to an embodiment for implementing a method for implementing a bank database migration and an embodiment for implementing a bank database migration apparatus, and the contents thereof are incorporated herein, and are not repeated here.

Fig. 10 is a schematic block diagram of a system configuration of a computer device 1000 according to an embodiment of the present application. As shown in fig. 10, the computer device 1000 may include a central processor 1001 and a memory 1002; the memory 1002 is coupled to the central processor 1001. Notably, this fig. 10 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the bank database migration function may be integrated into the central processor 1001. The central processor 1001 may be configured to control, among other things, the following:

Screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file to generate a target file;

And loading the target file into a target database.

In another embodiment, the bank database migration apparatus may be configured separately from the central processor 1001, for example, the bank database migration apparatus may be configured as a chip connected to the central processor 1001, and the bank database migration function is implemented under the control of the central processor.

As shown in fig. 10, the computer device 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, a power supply 1007. It is noted that the computer device 1000 need not include all of the components shown in FIG. 10; in addition, the computer device 1000 may further include components not shown in fig. 10, to which reference is made to the related art.

As shown in fig. 10, the central processor 1001, sometimes also referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, and the central processor 1001 receives input and controls the operation of the various components of the computer device 1000.

The memory 1002 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 1001 can execute the program stored in the memory 1002 to realize information storage or processing, and the like.

The input unit 1004 provides input to the central processor 1001. The input unit 1004 is, for example, a key or a touch input device. The power supply 1007 is used to provide power to the computer device 1000. The display 1006 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 1002 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, and the like. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. Memory 1002 may also be some other type of device. Memory 1002 includes a buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application/function storage 1022, the application/function storage 1022 for storing application programs and function programs or for executing a flow of operations of the computer apparatus 1000 by the central processor 1001.

The memory 1002 may also include a data store 1023, the data store 1023 for storing data such as contacts, digital data, pictures, sounds, and/or any other data used by a computer device. The driver store 1024 of the memory 1002 can include various drivers for the computer device for communication functions and/or for performing other functions of the computer device (e.g., messaging applications, address book applications, etc.).

The communication module 1003 is a transmitter/receiver 1003 that transmits and receives signals via an antenna 1008. A communication module (transmitter/receiver) 1003 is coupled to the central processor 1001 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 1003, such as a cellular network module, a bluetooth module, and/or a wireless lan module, etc., may be provided in the same computer device. The communication module (transmitter/receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and to receive audio input from the microphone 1010 to implement usual telecommunications functionality. The audio processor 1005 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 1005 is also coupled to the central processor 1001 so that sound can be recorded locally through the microphone 1010 and so that sound stored locally can be played through the speaker 1009.

The embodiment of the invention also provides a computer readable storage medium which stores a computer program for executing the bank database migration method.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims

1. A method for migrating a bank database, comprising:

loading the target file into a target database;

Screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated and the bank card number to be migrated from each data intermediate file to generate a target file, wherein the method comprises the following steps of:

Matching each client number in the data intermediate file with the client number to be migrated according to each data intermediate file, and screening to obtain data to be migrated corresponding to the client number to be migrated in the data intermediate file; generating a target file according to the data to be migrated corresponding to the client number to be migrated in each data intermediate file;

And/or matching the bank account corresponding to each client number in the data intermediate file with the bank account to be migrated according to each data intermediate file, and screening to obtain data to be migrated corresponding to the bank account to be migrated in the data intermediate file; generating a target file according to the data to be migrated corresponding to the bank account number to be migrated in each data intermediate file;

and/or matching the bank card number corresponding to each client number in the data intermediate file with the bank card number to be migrated for each data intermediate file, and screening to obtain data to be migrated corresponding to the bank card number to be migrated in the data intermediate file; and generating a target file according to the data to be migrated corresponding to the bank card number to be migrated in each data intermediate file.

2. The method as recited in claim 1, further comprising:

3. The method as recited in claim 1, further comprising:

4. The method of claim 1, wherein for each data intermediate file, matching the bank account corresponding to each client number in the data intermediate file with the bank account to be migrated, and screening to obtain data to be migrated corresponding to the bank account to be migrated in the data intermediate file, including:

for each data intermediate file:

extracting all bank accounts from the data intermediate file;

Adding a number to each bank account to obtain a first file; the number is a natural number sequence starting from 1; the serial numbers in the first file are sequenced according to the preset sequence of the bank account numbers, and a second file is obtained; the preset sequence comprises ascending sequence or descending sequence;

Matching the serial numbers of the bank accounts in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file after the serial numbers to obtain the data to be migrated corresponding to the bank accounts to be migrated in the data intermediate file; and the numbered data intermediate file is obtained by adding the number to the data intermediate file.

5. The method of claim 1, wherein for each data intermediate file, matching the bank card number corresponding to each client number in the data intermediate file with the bank card number to be migrated, and filtering to obtain data to be migrated corresponding to the bank card number to be migrated in the data intermediate file, comprising:

for each data intermediate file:

Extracting all bank card numbers from the data intermediate file;

adding a number to each bank card number to obtain a first file; the number is a natural number sequence starting from 1; the serial numbers in the first file are sequenced according to the preset sequence of the bank card numbers, and a second file is obtained; the preset sequence comprises ascending sequence or descending sequence;

Matching the serial number of the bank card number in the fourth file with the data to be migrated corresponding to different serial numbers in the data intermediate file after the serial number to obtain the data to be migrated corresponding to the bank card number to be migrated in the data intermediate file; and the numbered data intermediate file is obtained by adding the number to the data intermediate file.

6. The method of claim 1, wherein screening data to be migrated associated with each customer number to be migrated, the bank account number to be migrated, and the bank card number to be migrated from each data intermediate file, generating a target file, comprises:

7. The method as recited in claim 1, further comprising:

loading the target file into a target database, comprising:

8. A bank database migration apparatus, comprising:

the target file loading module is used for loading the target file to the target database;

the target file generation module is specifically configured to: matching each client number in the data intermediate file with the client number to be migrated according to each data intermediate file, and screening to obtain data to be migrated corresponding to the client number to be migrated in the data intermediate file; generating a target file according to the data to be migrated corresponding to the client number to be migrated in each data intermediate file;

9. The apparatus as recited in claim 8, further comprising:

The data table to be migrated determining module is used for:

10. The apparatus as recited in claim 8, further comprising:

the client number configuration module to be migrated is used for:

11. The apparatus of claim 8, wherein the object file generation module is specifically configured to:

for each data intermediate file:

extracting all bank accounts from the data intermediate file;

12. The apparatus of claim 8, wherein the object file generation module is specifically configured to:

for each data intermediate file:

Extracting all bank card numbers from the data intermediate file;

13. The apparatus of claim 8, wherein the object file generation module is specifically configured to:

14. The apparatus as recited in claim 8, further comprising:

the disk partition information determining module is used for:

the target file loading module is specifically configured to:

15. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 7 when executing the computer program.

16. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program for executing the method of any one of claims 1 to 7.