CN114936915B - Data processing method, device, electronic equipment and storage medium - Google Patents
Data processing method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114936915B CN114936915B CN202210190295.8A CN202210190295A CN114936915B CN 114936915 B CN114936915 B CN 114936915B CN 202210190295 A CN202210190295 A CN 202210190295A CN 114936915 B CN114936915 B CN 114936915B
- Authority
- CN
- China
- Prior art keywords
- target
- data
- file
- information
- data dictionary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 26
- 238000000034 method Methods 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 15
- 238000007405 data analysis Methods 0.000 abstract description 5
- 230000008859 change Effects 0.000 description 16
- 230000010354 integration Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000000926 separation method Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 208000024780 Urticaria Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 229910052697 platinum Inorganic materials 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013518 transcription Methods 0.000 description 1
- 230000035897 transcription Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/194—Calculation of difference between files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- General Engineering & Computer Science (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a data processing method, a data processing device, electronic equipment and a storage medium, and relates to the field of computers, in particular to the field of data analysis and intelligent search. The specific implementation scheme is as follows: acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by the downstream system based on the target class data.
Description
Technical Field
The present disclosure relates to the field of computer technology, and in particular, to a data processing method, apparatus, electronic device, and storage medium in the fields of data analysis and intelligent search.
Background
Currently, in the business processing process of a bank, a silver-colored union provides a data dictionary for a server of the bank, and a bank data analysis person needs to manually analyze the data dictionary to a data warehouse for analysis in a downstream system.
Disclosure of Invention
The present disclosure provides a method, apparatus, electronic device, and storage medium for data processing.
According to an aspect of the present disclosure, a data processing method is provided. The method comprises the following steps: acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by the downstream system based on the target class data.
According to another aspect of the present disclosure, another data processing apparatus is also provided. The device comprises: the first acquisition unit is used for acquiring a data dictionary, wherein the data dictionary is used for representing information of the bank card to be analyzed; the second acquisition unit is used for acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; and the generation unit is used for generating the target file analyzed by the downstream system based on the target category data.
According to another aspect of the present disclosure, an electronic device is also provided. The electronic device may include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing methods of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the data processing method of the embodiments of the present disclosure.
According to another aspect of the present disclosure, there is also provided a computer program product, which may comprise a computer program which, when executed by a processor, implements the data processing method of the embodiments of the present disclosure.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a distributed change bank card core system data dictionary automation resolution apparatus for instability features in accordance with an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 4 is a block diagram of an electronic device of a data processing method according to an embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a flow chart of a data processing method according to an embodiment of the present disclosure. As shown in fig. 1, the method may include the steps of:
step S102, a data dictionary is obtained, wherein the data dictionary is used for representing information of the bank card to be analyzed.
In the technical solution provided in the above step S102 of the present disclosure, the server of the bank may acquire a data dictionary from the union pay cloud, where the data dictionary is used to represent information of a bank card to be analyzed, and may be a core data dictionary file of the bank card, and may include file name information, update frequency information, incremental programming information, primary key information, field information, and the like, which is not limited herein specifically.
Optionally, the Unionpay cloud can synchronize the core data dictionary file of the bank card to a server (banking machine) of the bank at regular time so as to achieve the purpose of acquiring the data dictionary by the server of the bank.
Alternatively, the bank card may be a credit payment instrument, which may be a credit payment instrument approved for release to society by a financial institution (e.g., commercial bank, etc.), having all or part of the functions of consuming credit, settling accounts for transfers, accessing cash, etc., and may include: credit cards (credit cards) and digital bank cards, the credit cards can be divided into credit cards which are given to a cardholder by an issuing bank, the credit cards which can be paid after the cardholder consumes in the credit line and the cardholder can firstly pay a spare money with a certain amount according to the bank requirement, and when the spare money is not paid enough, the spare money can be overdrawn in the credit line regulated by the issuing bank; the digital bank card may be a flash card or the like, and is not particularly limited herein.
Alternatively, the bank card of this embodiment may further include a debit card that can be transferred and withdrawn on a network or an automated teller machine, which can be functionally divided into transfer cards having transfer, cash access and consumption functions, use of special cards for special areas, special purposes and prepaid wallet type stored value cards; the cards can be classified into ordinary cards, gold cards and platinum cards according to grades, and domestic cards and international cards according to application ranges.
The preferred application scenario for this embodiment is a related data dictionary file for a credit card core system; the above-mentioned business system of the debit card is independent of the business system of the credit card (credit card), but both have respective core data dictionary files, wherein the business of accessing the debit card is relatively simple, business and analysis scene of the credit card (credit card) are complex, therefore, the bank card of this embodiment can include not only the credit card (credit card) and digital bank card, but also the debit card.
For example, the server may obtain a dictionary of data to be analyzed from other credit payment instruments such as credit cards, digital bank cards, debit cards, and the like.
Step S104, obtaining target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card.
In the technical solution provided in the above step S104 of the present disclosure, the union pay cloud analyzes the corresponding information in the acquired data dictionary to acquire the target category information of the bank card corresponding to the downstream system, where the downstream system may be a system used in an application scenario of the bank card, may be a system implemented by using a first relational database management system (mysql), a system implemented by using a second relational database management system (Oracle), a data integration system, and so on; the target category information may be used for creating category information, updating category information, and the like, and may be created category information and updating category information generated by a mysql generation program configured for the first relational database management system (mysql); creation category information and update category information generated by an Oracle generation program configured for a second relational database management system (Oracle) may also be used; a configuration file generated by a generator configured for the data integration system, and the like, it should be noted that, the above system and system generation program are not particularly limited, but are merely illustrative.
Optionally, a data dictionary is acquired, and information corresponding to a downstream system in the data dictionary is analyzed to acquire target category information of the bank card.
For example, a data dictionary is obtained and if the downstream system is a system implemented using a first relational database management system (mysql), creation category information and update category information are generated based on a first relational database management system generator.
Step S106, generating a target file analyzed by a downstream system based on the target class data.
In this embodiment, the information corresponding to the downstream system in the data dictionary is parsed to obtain the target class information of the bank card, and the target file analyzed by the downstream system is generated based on the target class data, where the target file may be a structured query language (structured query language, abbreviated as sql) for table construction of different databases, a table structure change sql, a configuration file required by the data integration system, and the like.
Optionally, the data dictionary is obtained, the information corresponding to the downstream system in the data dictionary is analyzed, the target class information of the bank card is obtained, the task distribution processing is performed on the corresponding information in the data dictionary based on the target class information, and the configuration files required by different target systems of the downstream system, for example, the table construction sql of different databases, the table structure change sql, the configuration files required by the data integration system and the like are generated.
For example, a data dictionary is obtained, and if the downstream system is a first relational database management system (mysql), creation category information and update category information are generated based on a first relational database management system generation program, and table creation sql statements and table update sql statements are generated based on the creation category information and the update category information.
For example, a data dictionary is acquired, and if the downstream system is a second relational database management system (Oracle), creation category information and update category information are generated based on a second relational database management system generation program, and a table creation sql statement and a table update sql statement are generated based on the creation category information and the update category information.
For example, a data dictionary is obtained, and if the downstream system is a data integration system, a configuration file or the like is generated based on the data integration system.
Acquiring a data dictionary through the steps S102 to S106, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by the downstream system based on the target class data. That is, according to the method and the device, the target category information is obtained after the corresponding information in the data dictionary is analyzed, and the target file is generated based on the target category information, so that a downstream system can directly analyze the target file, the data dictionary of the bank card can be automatically analyzed, the efficiency of processing the data of the bank card is improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
The above-described method of this embodiment is described in further detail below.
As an optional implementation manner, step S204, obtaining the target class information of the bank card from the data dictionary includes: the target category information is read from the data dictionary based on the target toolkit.
In this embodiment, the table file in the data dictionary is parsed based on the target toolkit to obtain target category information, wherein the target toolkit can be used to read an extension tool (xlrd) of a table in a computer programming language (Python) and can be used to read the table file.
Optionally, the target toolkit based on Python analyzes the table file of the data dictionary, and generates target category information required by different downstream target systems based on the analysis result of the table file.
Optionally, the target tool is called to obtain the position of the corresponding field in the table according to the predetermined analysis logic, such as the file name, the update frequency, the primary key, the field separation rule, the field name, the field type, the field length, the field remark and other information in the table, so as to obtain the target category information.
As an alternative embodiment, reading the target category information from the data dictionary based on the target toolkit includes: reading a plurality of category information from the data dictionary to the memory based on the target toolkit; and acquiring the target category information in the plurality of category information from the memory by using a controller corresponding to the target category information.
In this embodiment, a target tool package is called to analyze a table file in a data dictionary and transmit the table file to a memory, a controller corresponding to target class information is determined, the controller obtains the target class information from the memory according to a configuration file program required by different downstream systems developed in advance and places the target class information in a designated position, so that the purpose of obtaining target class information in a plurality of class information from the memory is achieved, wherein the controller corresponding to the target class information can be a downstream controller, and can be a creation class task controller and an update class task controller.
Optionally, the target toolkit is called to analyze a table file in the data dictionary and transmit the table file to the memory, the task distribution module starts the downstream controller, the creation task controller can be started, the creation task controller obtains target class information from the memory according to different configuration file programs required by different downstream systems developed in advance and put the target class information into a designated position, for example, the task distribution module starts the downstream controller, the controller controls the task to start, and when the downstream system is a first relational database management system, the controller obtains the target class information from the memory according to a first relational database management system configuration generating program.
Optionally, an update task controller may be started, where the update task controller obtains target class information from the memory according to a profile program required by different downstream systems developed in advance, and places the target class information to a specified target.
Alternatively, the create class task controller and the update class task controller may be started simultaneously, wherein. The create class task controller generates a create class task each time, and the update class task may generate a specific update file by a profile program required by the downstream system when checking that the previous version dictionary file exists and when there is a difference in comparison.
As an optional implementation, step S206, where the target category information includes update category task data, generating the target file analyzed by the downstream system based on the target category data includes: and updating the original file to be analyzed of the downstream system based on the update task data to obtain the target file.
In this embodiment, when update task data is obtained, the update task data may be compared with an original file, and when there is a difference in comparison, the original file is updated to obtain a target file, where the original file may be a data dictionary file of a previous version that is smaller than the current version; the update class task data may be table data of a newly acquired data dictionary.
Optionally, the data dictionary has time sequence numbers on file names, the data dictionary files of the previous version smaller than the current version are obtained by sorting according to the data dictionary names, the original file is automatically loaded after the downstream service system detects that the update type latest configuration file is generated, the update type latest configuration file is compared with the original file, and when the comparison has a difference, the original file is updated to obtain the target file.
As an alternative embodiment, the method further comprises: comparing the target category information with the original category information of an original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing the original information of the bank card to be analyzed; and generating update type task data based on the comparison result in response to the comparison result indicating that the target type information is different from the original type information.
In the embodiment, the target class information is compared with the original class information of the original data dictionary to obtain a comparison result, and if the difference exists between the target class information and the original class information of the original data dictionary, the comparison result with the difference is responded to generate the update class task data.
Optionally, comparing the original information of the bank card to be analyzed with the target category information to obtain a comparison result, and if the difference exists between the target category information and the original information, generating update type task data.
Optionally, the embodiment may further set a change comparison record generating program, and when there is a difference between the target category information and the original information, the change comparison record generating program is triggered to automatically generate a change record file, where the change record may include an update date, an update content, and the like, and it is to be noted that the change record content may be set according to actual needs, which is not specifically described herein.
As an alternative embodiment, the method further comprises: an update message is sent to a downstream system, wherein the update message is used for indicating that an original file is updated to be a target file; and receiving a request sent by the downstream system based on the update message, and responding to the request and sending the target file to the downstream system.
In this embodiment, the update task obtains update task data from the memory according to the configuration file program required by different downstream systems developed in advance to generate a corresponding target file, and places the target file in the designated directory, and when the original file is updated to the target file, the update task manager issues an update message to the downstream system, receives a request sent by the downstream system based on the update message, and responds to the request, and issues the target file to the downstream system.
Optionally, the update task obtains update task data from the memory according to the configuration file program required by different downstream systems developed in advance to generate a corresponding file, and the corresponding file is placed at a designated position, and after the downstream service system detects that the update is the target file, the corresponding file is automatically loaded, so that the purpose of issuing the target file to the downstream system is achieved.
As an alternative embodiment, the target class information includes creation class task data, and generating the target file analyzed by the downstream system based on the target class data includes: a target file is created for analysis by the downstream system based on the creation class task data.
In this embodiment, the target class information includes creation class task data, which may be table data, and a target file analyzed by a downstream system is created based on the creation class task data, where the target file may create sql statements for a table generated by a first relational database management system (mysql) configuration generator or may create sql statements for a table generated by a second relational database management system (Oracle) configuration generator.
It should be noted that, mysql and Oracle have different actual table construction statement and field types, different creation/update statement needs to be generated, and the mysql configuration generation program and Oracle configuration generation program will not be started at the same time, and can be started in configuration according to the requirement of the downstream system.
Optionally, the task distribution module may start a downstream creation class task controller, where the creation class task obtains creation class task data from the memory according to a profile program (for example, mysql configuration generator, oracle configuration generator) required by different downstream systems developed in advance, and generates a target file analyzed by the downstream systems.
As an optional implementation manner, step S202, obtaining the data dictionary includes: and reading the data dictionary with the time stamp closest to the current time from the first storage position according to the target time period.
In this embodiment, the data dictionary with the timestamp closest to the current time is read from the first storage location according to the target time period, where the first storage location may be a designated directory, and the target time period may be set according to the actual requirement, may be 1 or 2, and is not specifically limited herein.
Optionally, the latest data dictionary in the first storage location (designated directory) is read, and the data dictionary with the timestamp closest to the current time is read from the first storage location according to the target time period, for example, when judging whether the latest day file exists, the latest day file may be detected in a T-1 manner, if the task is triggered at the month 28 of 2021, the file with the suffix of the data dictionary of the day 27 of 2021 and the date of 2021 is detected, and the number is only for illustration and not limited in particular.
As an alternative embodiment, the method further comprises: storing the target file to a second storage location, wherein the target file is retrieved from the second storage location by the downstream system.
In this embodiment, a data dictionary is acquired, target class information of the bank card is acquired from the data dictionary, a target file analyzed by the downstream system is generated based on the target class data, the target file is stored in the second storage location, and the downstream system acquires the target file from the second storage location.
Optionally, after reading the latest dictionary, the parsing device parses the content of the dictionary file into the memory using the target toolkit, calls the target toolkit to obtain target category information, such as file name, update frequency, primary key, field separation rule, field name, field type, field length, field remark, and the like, according to the predetermined parsing logic, then stores the target file in the memory in a second storage location (for example, in the form of a global shared variable, the data structure may be a set of paired structures), and the downstream system may directly obtain the target file from the second storage location.
In this embodiment, the data dictionary is used to represent information of the bank card to be analyzed by acquiring the data dictionary; acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; a target file is generated for analysis by the downstream system based on the target class data. That is, according to the method and the device, the target category information is obtained after the corresponding information in the data dictionary is analyzed, and the target file is generated based on the target category information, so that a downstream system can directly analyze the target file, the data dictionary of the bank card can be automatically analyzed, the efficiency of processing the data of the bank card is improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
The foregoing technical solutions of the embodiments of the present disclosure are further described by way of example with reference to the preferred embodiments.
At present, in banking card business of banks, most banks directly purchase a banking card core business system of a banked union, the card core system is hosted in a banked cloud, a bank data analysis department cannot directly access the banking card core business system, a general banked union synchronizes data to a server of the banks every day through a data exchange system, and the banks analyze the data to own data warehouse according to the data files for analysis.
In the processing process of the related technology, the Unionpay provides a data dictionary, the data dictionary relates to hundreds of tables, each table word Duan Duoze is hundreds, and dozens of tables are fewer, a bank data analysis person needs to copy fields one by one and modify the fields into table building sentences, table changing sentences and code configuration, and the Unionpay periodically updates the fields, but after each data dictionary update, the bank also needs to manually compare specific fields and modify table structures, configuration and codes.
Therefore, to solve the above-mentioned problems, the apparatus for automatically parsing data dictionary tables (Excel) based on the toolkit xlrd toolkit of the Python reading table in the embodiment of the present disclosure automatically generates relevant configuration files for table construction Sql and code development; under the condition of historical version, sql and change records of an update table are generated by comparing with the previous version data dictionary, so that development efficiency and accuracy are greatly improved, perceived data change can be automatically compared, and related change records are generated, so that full-flow automatic processing of the core system data of the subsequent silver-jointed card can be realized.
Optionally, the data integration system needs to perform different structural analyses according to different field lengths, separator types, field similarities and other conditions in excel, the data integration system code is abstract universal code, the code is not required to be modified, different tasks can be adapted through modification of the configuration, and the configuration required by the data integration system can be converted into the analysis logic
Fig. 2 is a schematic diagram of an automated analysis device for a data dictionary of a distributed change bank card core system according to an embodiment of the present disclosure, as shown in fig. 2, a banking association synchronizes a data dictionary file of a bank card core to a banking machine at regular time, and the automated analysis device for a data dictionary of a bank card system (hereinafter referred to as an analysis device) and the data dictionary file of a card core are deployed in the same server.
First, the daily timing task automatically triggers the analyzing device.
Alternatively, the timing task may automatically trigger the parsing means by a timer or the like.
And secondly, the analysis device reads the dictionary file with the latest appointed catalogue.
The analysis device responds to the trigger signal to judge whether the latest day file exists or not, and the analysis device can detect the latest day file according to a T-1 mode, for example, the trigger task of the day 28 of 12 in 2021 can detect the file with the suffix of the card core of the day 27 of 12 in 2021.
Optionally, the parsing device reads the latest dictionary file from the card core dictionary in response to the trigger signal, and obtains file name information, update frequency information, increment rule information, primary key information, field information, and the like.
And thirdly, analyzing the content into the memory by using xlrd of the dictionary file.
And the text analysis module in the analysis device analyzes the dictionary data based on xlrd after reading the latest dictionary, and distributes the table analysis result to the memory.
Optionally, the method xlrd is called to obtain the position of the corresponding field in the table according to the predetermined parsing logic, such as file name, update frequency, primary key, field separation rule, field name, field type, field length, field remark, etc., then the information is stored in the memory by using a global shared variable, the data structure is a set of opposite structure (map structure), and other program codes directly read the shared variable to obtain the required information.
And fourthly, the analysis device task distribution module starts a downstream controller, and mainly comprises a creation class task and an update class task.
Optionally, the task distribution module starts a downstream controller, and the controller configures a creation class task control and an update class task control, wherein the creation class task and the update class task are started simultaneously, the full creation class task is generated each time, and the update class task generates a specific update file only when the existence of the dictionary file of the previous version is checked and the difference is compared.
Optionally, the task distribution module configures the creation class task controller and the update class task controller.
And fifthly, the creation class task acquires form data from the memory according to the configuration file programs required by different downstream systems developed in advance to generate corresponding files and puts the files into the appointed target.
Optionally, according to the downstream application scenario, the task controller selects a required configuration file program, where the configuration creation class task control may include a mysql configuration generation program of the first relational database management system, an Oracle configuration generation program of the second relational database management system, and a data integration program configuration generation program.
Alternatively, a table creation sql statement may be generated based on a mysql configuration generator of the first relational database management system; creating an sql statement based on an Oracle configuration generator of the second relational database management system; the configuration file may be generated based on the data integration program configuration generation program.
Alternatively, the actual build statement and field type of mysql and Oracle are different, different create/update statements need to be generated, and an adapter extension can be added to databases such as data warehouse tools (hives), distributed databases (Doris).
And sixthly, the update task obtains form data from the memory according to the configuration file programs required by different downstream systems developed in advance to generate corresponding files and place the files in the appointed catalogue.
Optionally, according to the downstream application scenario, the task controller selects a required configuration file program, and the configuration update task controller may include a mysql configuration generation program of the first relational database management system, an Oracle configuration generation program of the second relational database management system, a data integration program configuration generation program, and a change comparison record generation program.
Alternatively, a table creation sql statement may be generated based on a mysql configuration generator of the first relational database management system; creating an sql statement based on an Oracle configuration generator of the second relational database management system; the generating program can be configured based on the data integration program to generate a configuration file; the change log file may be generated based on a change contrast log generation program.
And seventhly, the downstream service system automatically loads the corresponding file to change after detecting that the update type latest configuration file is generated.
Optionally, the data dictionary of the previous version is changed, the data dictionary has time sequence numbers on file names, and the data dictionary files of the previous version smaller than the current version can be obtained by sorting according to the data dictionary names.
Optionally, after generating the update file, the downstream system sorts according to the data dictionary names to obtain the data dictionary file of the previous version smaller than the current version, and changes the data dictionary file of the previous version.
According to the embodiment of the disclosure, the xlrd toolkit based on Python automatically analyzes the Excel file of the data dictionary of the bank card core system, corresponding information in the data dictionary is analyzed to generate configuration files required by different downstream target systems, such as a table building sql of different databases, a table structure is changed sql, and the configuration files required by the data integration system are changed to the full-flow automation of the adaptation of the data integration task by solving the original manual transcription comparison and duplication field mode, so that the technical problems of low development efficiency and processing efficiency of data caused by incapability of automatization of flow are solved, and the technical effects of improving the development efficiency and the processing efficiency of the data are realized.
The embodiment of the disclosure also provides a data processing device for executing the data processing method of the embodiment shown in fig. 1.
Fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in fig. 3, the data processing apparatus 30 may include: a first acquisition unit 31, a second acquisition unit 32, and a generation unit 33.
A first acquiring unit 31 for acquiring a data dictionary for representing information of the bank card to be analyzed.
The second obtaining unit 32 is configured to obtain, from the data dictionary, target class information of the bank card, where the target class information corresponds to a downstream system, and the downstream system is a system used in an application scenario of the bank card.
A generation unit 33 for generating a target file analyzed by the downstream system based on the target class data
Optionally, the second acquisition unit 32 includes: and the first reading module is used for reading the target category information from the data dictionary based on the target toolkit.
Optionally, the first reading module includes: the first acquisition sub-module is used for reading out a plurality of category information from the data dictionary to the memory based on the target tool kit; and acquiring the target category information in the plurality of category information from the memory by using a controller corresponding to the target category information.
Alternatively, the generating unit 33 includes: and the first updating module is used for updating the original file which needs to be analyzed by the downstream system based on the updating task data to obtain the target file.
Optionally, the first updating module includes: the first comparison sub-module is used for comparing the target category information with the original category information of the original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing the original information of the bank card to be analyzed; and generating update type task data based on the comparison result in response to the comparison result indicating that the target type information is different from the original type information.
Alternatively, the generating unit 33 includes: the second updating module is used for issuing an updating message to the downstream system, wherein the updating message is used for indicating that the original file is updated to be the target file; and receiving a request sent by the downstream system based on the update message, and responding to the request and sending the target file to the downstream system.
Alternatively, the generating unit 33 includes: and the first creating module is used for creating the target file analyzed by the downstream system based on the creation class task data.
Alternatively, the first acquisition unit 31 includes: and the first reading unit is used for reading the data dictionary with the time stamp closest to the current time from the first storage position according to the target time period.
Optionally, the apparatus further comprises: and the storage unit is used for storing the target file to a second storage position, wherein the target file is acquired from the second storage position by a downstream system.
In the device of the disclosed embodiment, a data dictionary is acquired through a first acquisition unit, wherein the data dictionary is used for representing information of a bank card to be analyzed; acquiring target class information of the bank card from the data dictionary through a second acquisition unit, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card; the generation unit is used for generating the target file analyzed by the downstream system based on the target type data, so that the efficiency of processing the data of the bank card is improved, and the technical problem of low efficiency of processing the data of the bank card is solved.
In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
Embodiments of the present disclosure provide an electronic device that may include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing methods of the embodiments of the present disclosure.
Optionally, the electronic device may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.
According to an embodiment of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the data processing method of the embodiments of the present disclosure.
Alternatively, in the present embodiment, the above-described nonvolatile storage medium may be configured to store a computer program for performing the steps of:
s1, acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed;
S2, acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
S3, generating a target file analyzed by a downstream system based on the target class data.
Alternatively, in the present embodiment, the non-transitory computer readable storage medium described above may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of:
s1, acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed;
S2, acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
S3, generating a target file analyzed by a downstream system based on the target class data.
Fig. 4 is a block diagram of an electronic device of a data processing method according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 4, the apparatus 400 includes a computing unit 401 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In RAM403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM402, and RAM403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
Various components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, etc.; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the respective methods and processes described above, for example, a method data processing method. For example, in some embodiments, the method data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When a computer program is loaded into RAM403 and executed by computing unit 401, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the data processing method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (11)
1. A data processing method, comprising:
Acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed;
acquiring target class information of the bank card from the data dictionary, wherein the target class information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
Generating a target file for analysis by the downstream system based on the target class data;
The method for acquiring the target category information of the bank card from the data dictionary comprises the following steps: reading a plurality of category information from the data dictionary to a memory based on the target toolkit; and acquiring target category information in the plurality of category information from the memory by using a controller corresponding to the target category information.
2. The method of claim 1, wherein the target category information includes update category task data, generating a target file for analysis by the downstream system based on the target category data includes:
and updating the original file to be analyzed by the downstream system based on the update task data to obtain the target file.
3. The method of claim 2, further comprising:
Comparing the target category information with original category information of an original data dictionary to obtain a comparison result, wherein the original data dictionary is used for representing the original information of the bank card to be analyzed;
and generating the update type task data based on the comparison result if the comparison result shows that the target type information and the original type information are different.
4. The method of claim 2, further comprising:
issuing an update message to the downstream system, wherein the update message is used for indicating that the original file is updated to the target file;
and receiving a request sent by the downstream system based on the update message, and responding to the request and sending the target file to the downstream system.
5. The method of claim 1, wherein the target category information includes creation class task data, generating a target file for analysis by the downstream system based on the target category data includes:
creating the target file for analysis by the downstream system based on the creation class task data.
6. The method of any of claims 1-5, the obtaining the data dictionary comprising:
and reading the data dictionary with the time stamp closest to the current time from the first storage position according to the target time period.
7. The method of any one of claims 1 to 5, further comprising:
And storing the target file to a second storage location, wherein the target file is acquired from the second storage location by the downstream system.
8. A data processing apparatus comprising:
the first acquisition unit is used for acquiring a data dictionary, wherein the data dictionary is used for representing information of a bank card to be analyzed;
the second acquisition unit is used for acquiring target category information of the bank card from the data dictionary, wherein the target category information corresponds to a downstream system, and the downstream system is a system used in an application scene of the bank card;
A generation unit configured to generate a target file analyzed by the downstream system based on the target class data;
Wherein the second obtaining unit is configured to obtain the target category information of the bank card from the data dictionary by: reading a plurality of category information from the data dictionary to a memory based on the target toolkit; and acquiring target category information in the plurality of category information from the memory by using a controller corresponding to the target category information.
9. An electronic device, comprising:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.
11. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210190295.8A CN114936915B (en) | 2022-02-28 | 2022-02-28 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210190295.8A CN114936915B (en) | 2022-02-28 | 2022-02-28 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114936915A CN114936915A (en) | 2022-08-23 |
CN114936915B true CN114936915B (en) | 2024-09-06 |
Family
ID=82861820
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210190295.8A Active CN114936915B (en) | 2022-02-28 | 2022-02-28 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114936915B (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862604A (en) * | 2021-04-25 | 2021-05-28 | 腾讯科技(深圳)有限公司 | Card issuing organization information processing method, device, equipment and storage medium |
CN113902415A (en) * | 2021-10-26 | 2022-01-07 | 中国工商银行股份有限公司 | Financial data checking method and device, computer equipment and storage medium |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1252580A1 (en) * | 1999-10-29 | 2002-10-30 | Computer Science Corporation | Business transaction processing systems and methods |
US20070061245A1 (en) * | 2005-09-14 | 2007-03-15 | Jorey Ramer | Location based presentation of mobile content |
CN102932419B (en) * | 2012-09-25 | 2016-02-10 | 浙江图讯科技股份有限公司 | A kind of data-storage system for the safety production cloud service platform towards industrial and mining enterprises |
US20200098053A1 (en) * | 2018-09-26 | 2020-03-26 | Intuit Inc. | Method and system for user data driven financial transaction description dictionary construction |
CN109543690B (en) * | 2018-11-27 | 2020-04-07 | 北京百度网讯科技有限公司 | Method and device for extracting information |
CN110704462B (en) * | 2019-09-06 | 2024-09-17 | 中国平安财产保险股份有限公司 | Data change notification method, electronic device, computer apparatus, and storage medium |
CN112069773A (en) * | 2020-07-23 | 2020-12-11 | 北京三快在线科技有限公司 | Data processing system, method, apparatus, electronic device, and computer-readable medium |
CN112925914B (en) * | 2021-03-31 | 2024-03-15 | 携程旅游网络技术(上海)有限公司 | Data security grading method, system, equipment and storage medium |
CN113269551A (en) * | 2021-05-25 | 2021-08-17 | 北京金山云网络技术有限公司 | Medical payment method, device and system and electronic equipment |
-
2022
- 2022-02-28 CN CN202210190295.8A patent/CN114936915B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862604A (en) * | 2021-04-25 | 2021-05-28 | 腾讯科技(深圳)有限公司 | Card issuing organization information processing method, device, equipment and storage medium |
CN113902415A (en) * | 2021-10-26 | 2022-01-07 | 中国工商银行股份有限公司 | Financial data checking method and device, computer equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN114936915A (en) | 2022-08-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109034988B (en) | Accounting entry generation method and device | |
CN111382279A (en) | Order examination method and device | |
CN114579104A (en) | Data analysis scene generation method, device, equipment and storage medium | |
CN114021156A (en) | Method, device and equipment for organizing vulnerability automatic aggregation and storage medium | |
CN113836314A (en) | Knowledge graph construction method, device, equipment and storage medium | |
CN111930819A (en) | ETL script generation method and device | |
CN114092948B (en) | Bill identification method, device, equipment and storage medium | |
CN114936915B (en) | Data processing method, device, electronic equipment and storage medium | |
CN114064925A (en) | Knowledge graph construction method, data query method, device, equipment and medium | |
CN115525721A (en) | Data synchronization method, device, equipment and storage medium | |
CN111078784A (en) | Report automatic recommendation system, method and computer system | |
CN116955856A (en) | Information display method, device, electronic equipment and storage medium | |
CN117033431A (en) | Work order processing method, device, electronic equipment and medium | |
CN114139798A (en) | Enterprise risk prediction method and device and electronic equipment | |
CN114661918A (en) | Knowledge graph construction method and device, storage medium and electronic equipment | |
CN115167855A (en) | Front-end page generation method, device and equipment applied to matching transaction system | |
CN114676266A (en) | Conflict identification method, device, equipment and medium based on multilayer relation graph | |
CN114860872A (en) | Data processing method, device, equipment and storage medium | |
CN112380321A (en) | Primary and secondary database distribution method based on bill knowledge graph and related equipment | |
CN114461665A (en) | Method, apparatus and computer program product for generating a statement transformation model | |
CN112783918A (en) | Search method, search apparatus, electronic device, storage medium, and program product | |
CN115082179A (en) | Data processing method, device, equipment and storage medium | |
CN114547085B (en) | Data processing method and device, electronic equipment and storage medium | |
US11150955B2 (en) | Generation and usage of a task-based programming platform | |
CN118095270B (en) | Construction method and device of logic analysis graph, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |