KR20170031750A - Managing lineage information - Google Patents

Abstract

Managing lineage information used by data lineage module 115 includes receiving lineage information indicative of at least one lineage relationships between at least two data processing programs and at least two logical data sets 810, 1110, ; Receiving at least one runtime artifact 429, each runtime artifact comprising information relating to a previous execution of a data processing program of the at least two data processing programs; And analyzing the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information.

Description

Management of lineage information {MANAGING LINEAGE INFORMATION}

This application claims priority to U.S. Serial No. 62 / 026,228 filed on July 18, 2014.

This specification relates to managing parameter sets.

In a data processing system it is often desirable for certain types of users to be able to access reports of the lineage of data as it passes through the system. Very commonly, among many uses, these "data lineage" reports can be used to reduce risk, identify regulatory compliance obligations, simplify business processes, and protect data. Data lineage reports should be accurate and complete.

An object of the present invention is to provide a method for managing lineage information, software for managing geneage information, and a computing system for managing geneage information.

In one aspect, managing sets of lineage information parameter values, which generally reflect relationships between instances of generic computer programs that have been instantiated using these sets of parameter values, enables the generation of more accurate and complete data lineage reports do.

In one aspect, in general, a method for managing lineage information

Receiving lineage information representative of at least one lineage relationships between at least two data processing programs and at least two logical data sets; Receiving at least one runtime artifacts, each runtime artifact comprising information relating to a previous execution of a data processing program of the at least two data processing programs; And analyzing the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information.

The sides may include one or more of the following features.

The at least one candidate modifications include a candidate modification that adds a new indirect genealogical relationship between the data processing program of the at least two data processing programs and the logical data set of the at least two logical data sets.

The at least one candidate modifications include a first candidate modification that adds a new direct lineage relationship between the data processing program of the at least two data processing programs and the logical data set of the at least two logical data sets.

Wherein analyzing the runtime artifact and the genealogy information comprises analyzing logs of previous runs of the at least two data processing programs to determine physical data sets to be read or written by the at least two data processing programs .

Analyzing the runtime artifact and the genealogy information further comprises identifying two distinct sets of logical data of the at least two logical data sets represented in the genealogical information and associated with the same physical data set.

The first candidate modification includes generating a new lineage relationship between the two separate sets of logical data.

The first candidate modification includes merging the two separate sets of logical data into a new combined logical data set.

Each data processing program of the at least two data processing programs is an instance of a generic data processing program instantiated according to a set of at least one parameter values.

Wherein analyzing the at least one runtime artifact and the genealogy information comprises comparing at least one logs of previous runs of the first data processing program of the at least two data processing programs with a first set of at least one parameter values Analyzing to determine a first set of parameters to be used in a first instantiation of the first data processing program, selecting at least some parameters from the first set of parameters, Determining that the instantiation is not represented in the genealogical information based on the generic version of the first data processing program and the at least some parameters.

Wherein selecting at least some parameters from the first set of parameters comprises selecting parameters based on information received from a user.

Wherein selecting at least some parameters from the first set of parameters comprises selecting parameters based on at least one predefined rule.

The first rule of the at least one predefined rule specifies that parameters having parameter values in date form are excluded from the selected parameters.

The first rule of the at least one predefined rule specifies that a parameter having a parameter value converted into logic of the generic data processing program is included in the selected parameters.

Wherein the at least one candidate modifications to the genealogy information include a first candidate modification that adds a new genealogical relationship between the first data processing program of the at least two data processing programs and the logical data set of the at least two logical data sets .

In another aspect, generally, software for managing lineage information is stored in a non-transitory manner on a computer-readable medium, the software being programmed such that the computing system has at least two data processing programs and at least two Receive lineage information representative of at least one lineage relationships between logical data sets; To receive at least one runtime artifacts, each runtime artifact comprising information relating to a previous run of a data processing program of the at least two data processing programs; And

And to analyze the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information.

In another aspect, generally, a computing system for managing lineage information includes at least one data processing program, and at least one of a plurality of data processing programs and at least one An input device or port configured to receive runtime artifacts, each runtime artifact comprising information relating to a previous execution of a data processing program of the at least two data processing programs; And at least one processor configured to analyze the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information.

In another aspect, generally, a computing system for managing lineage information includes at least one data processing program, and at least one of a plurality of data processing programs and at least one Means for receiving runtime artifacts, each runtime artifact comprising information relating to a previous execution of a data processing program of the at least two data processing programs; And means for analyzing the at least one runtime artifacts and the lineage information to determine at least one candidate modifications to the lineage information.

The aspects may include one or more of the following advantages.

The data lineage reports generated using the enhanced set of existing parameter sets, by discovering the parameter sets using the methods described herein and using the found parameter sets to augment the existing set of parameter sets, It more accurately represents true data lineage. In particular, portions of the data lineage for data processing systems that may have been overlooked previously are included in the data lineage report.

In some instances, the results of the parameter set discovery methods may also be used to augment log entries of the executions of instances of the computer program (i. E. Reinforce log entries with information about discovered parameter sets). Enhanced log entries can be advantageously used to verify that logical connections between computer programs and / or data sets correspond to physical connections. These verification results ensure that the data lineage presented to the user shows accurate lineage relationships between computer programs and their inputs and outputs.

Other features and advantages of the invention will be apparent from the following description, and from the claims.

Figure 1 is a block diagram of a system for discovery of parameter sets.
Figure 2 is a dataflow graph containing sub-graphs and their associated parameter sets.
FIG. 3 shows the execution time configuration of the data flow graph of FIG.
Figure 4 is a static analysis configuration of the dataflow graph of Figure 2;
Figure 5 is a sequence of methods for the discovery of parameter sets.
Figure 6 is a first part of an exemplary operation of a method for discovery of parameter sets.
Figure 7 is a later part of an exemplary operation of a method for discovery of parameter sets.
8 is an example data flow graph including a first subgraph and a second subgraph.
Figure 9 shows the dataflow graph of Figure 8 with logical data sets resolved into physical data sets.
Figure 10 shows a data lineage report for the dataflow graph of Figure 8;
11 is an exemplary dataflow graph that includes a first subgraph and a second subgraph and has redundant logical data sets.
Figure 12 shows an example data flow graph of Figure 11 with logical data sets interpreted as physical data sets.
13 shows a data lineage report that includes a data lineage break for the data flow graph of FIG.
Figure 14 shows a first technique for mitigating the impact of data lineage breaks in a data lineage report.
15 shows a second technique for mitigating the effect of data lineage breaks in the data lineage report.
Figure 16 shows a third technique for mitigating the impact of data lineage breaks in the data lineage report.
Figure 17 shows a fourth technique for mitigating the impact of data lineage breaks in the data lineage report.
18 shows a fifth technique for mitigating the effects of data lineage breaks in the data lineage report.
19 illustrates a sixth technique for mitigating the effects of data lineage breaks in the data lineage report.

FIG. 1 illustrates an example of a data processing system 100 in which the parameter set discovery techniques described herein may be used. The system includes dataflow graphs 117 (see FIG. 1) that include vertices (representing data processing components or datasets) connected by directional links (representing work elements or data flows) between vertices in some implementations ), Which is a system for developing an application. For example, such an environment is described in greater detail in U.S. Publication No. 2007/0011668 entitled " Managing Parameters for Graph-Based Applications ", which is incorporated herein by reference. A system for performing such graph-based calculations is described in U.S. Patent 5,966,072, titled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS", which is incorporated herein by reference. The dataflow graphs 117 made in accordance with the system can be used to send information to individual processes represented by graph components and to obtain information from individual processes, to move information between the processes, Provides methods for defining the execution order for processes. The system can be used in any of the available ways (e.g., communication paths along the graph links can use TCP / IP or UNIX domain sockets or use shared memory to pass data between processes) And an algorithm for selecting an inter-process communication method. The dataflow graphs 117 generated by the developer 120 using the development environment 118 are accessible to the development environment 118 for subsequent use by other modules of the system 100. [ May be stored in the data storage system 116.

The execution environment 104 includes a parameter resolution module 106 and an execution module 112. The execution environment 104 may be hosted on one or more general purpose computers under the control of a suitable operating system, such as, for example, a UNIX operating system version. For example, the execution environment 104 may be a local (e.g., multiprocessor system such as a symmetric multiprocessing (SMP) computer) or a locally distributed (e.g., multiple processors coupled into clusters, (CPUs) or processors (MPUs), which are remote or remote distributed (e.g., local area network (LAN) and / or wide area network (WAN) And a multi-node parallel computing environment including configurations of computer systems using cores.

The parametric analysis module 106 receives the specifications of the dataflow graphs 117 from the data storage system 116 and receives the dataflow graphs 117 for execution by the execution module 112. [ (As described in more detail below) to interpret the parameters for the dataflow graphs 117. The execution module 112 receives the prepared data flow graphs 117 from the parameter interpretation module 106 and processes the data from the data source 102 and uses them to generate output data 114 do. The output data 114 may be stored again in the data storage system 116 accessible to the data source 102 or in the execution environment 104, or otherwise used. Generally, the data source 102 may include one or more data sources, such as connections to storage devices or online data streams, each of which may be in a variety of formats (e.g., a database table, a spreadsheet files), flat text files, or raw formats used by the mainframe).

The storage devices providing the data source 102 may be stored on a storage medium (e.g., hard drive 108) connected to a computer hosting the execution environment 104, for example, , Or may be local to a remote system (e. G., A < / RTI > main) that communicates with a computer hosting the execution environment 104 via a remote connection (e. G., Provided by a cloud computing infrastructure) Frame 110) and remotely to the execution environment 104.

The system 100 also includes a metadata environment module 119 that is accessible to enterprise users 121 (e.g., data architects or business users). The metadata environment module 119 is configured to generate the data flow graphs 117 (or the metadata that characterizes them and the input and output data sets they refer to to generate the data lineage for the data flow graphs 117) And a data lineage module 115 for processing the data. The enterprise user 121 can view the data lineage for reasons such as verification of the data flow graphs 117 and compliance checking. The data lineage information for a particular data item (e.g., a data set or a field in a data set) is based on a dependency that arises due to processing performed by the data processing system, and the term " data lineage " Generally refers to a set of other related data items and processing entities that consume or generate those data items. A data lineage report (also referred to as a data lineage diagram) may include a graphical representation of a data lineage in the form of a graph having nodes representing data items and processing entities and links representing dependencies between them. Some systems that can generate and display data lineage reports are end-to-end data lineage from the last data sources in the upstream end to the final data in the downstream end. ) Can be displayed automatically. Nodes on the upstream path from a particular data item are sometimes referred to as "dependencies" for that data item and nodes on the downward path from a particular data item are sometimes referred to as "impacts" on that data item. While " data lineage " is sometimes used to denote only upward dependency, as used herein, a " data lineage " may indicate one or both of an upward dependency and / or a downward impact, as appropriate to the particular situation.

One. Dataflow graph overview

Referring to FIG. 2, an example of a dataflow graph 217 generated using the execution environment 118 of FIG. 1 includes a first subgraph 202 named gather.mp and a second subgraph 202 named process.mp. 2 subgraph (204).

The first subgraph 202 receives as input a first logical data set DS1 206 and a second logical data set DS2 208 and generates a first logical data set DS1 206 and a second logical data set DS2 208 from the first and second logical data sets 206, And records the processing result in the third logical data set DS3 (210). The second subgraph 204 receives as inputs a fourth logical data set DS4 212 (indicating the same physical file as the third logical data set 210) And records the result of the processing in the table 214. [

Each of the four sets of logical data 206, 208, 210, 212 is associated with a parameterized path that is interpreted as a path to a physical file at runtime. In particular, the first logical data set 206 is identified using the parameterized path / $ { FEED} / inv_ $ { DATE} .dat , and the second logical data set 208 is identified using the parameterized path / $ { FEED} / cust _ $ {DATE} .dat , and the third logical data set 210 is identified using the parameterized path /trans_${DATE}.dat , and the fourth logical data set 212 ) Is identified using the parameterized path /trans_${DATE}.dat .

The first subgraph 202 receives two parameters P1 = FEED and P2 = DATE as arguments and feeds FEED and FEED of the parameterized paths to the values of the received FEED and DATE parameters, as described in more detail below. To interpret the path to each physical location of the first logical data set 206, the second logical data set 208 and the third logical data set 210 by replacing the DATE placeholder, Lt; / RTI > Also, the first subgraph 202 includes a " static analysis " value for the DATE parameter. As will be described in more detail below, the static analysis value for the DATE parameter is used as a parameter value during the static analysis of the dataflow graph 217 (i.e., when the data lineage of the dataflow graph 217 is determined) It is an indicator value.

Similarly, the second subgraph 104 receives the single parameter P1 = DATE and replaces the DATE placeholder of the parameterized path for the fourth set of logical data 212 with the value of the received DATE parameter, And uses it to interpret the path to the physical location of the data set 212. Also, the second subgraph 204 includes a " static analysis " value for the DATE parameter. As will be described in more detail below, the static analysis value for the DATE parameter is used as a parameter value during the static analysis of the dataflow graph 217 (i.e., when the data lineage of the dataflow graph 217 is determined) It is an indicator value.

Because the operation of the dataflow graph 217 and its subgraphs depends on the parameters it receives, the dataflow graph and its subgraphs are sometimes referred to as "generic" dataflow graphs or "generic ) "It is also called a computer program.

1.1 Parameters

In general, the above-mentioned parameters may be designated with a "design time" parameter or a "run time" parameter. In addition to being used for path analysis as described above, design time parameters affect the logical operation of the associated dataflow graph. Conversely, run-time parameters are provided in the work-by-work graph and do not affect the logical operation of the graph. In some instances, the logical operations of the dataflow graph refer to both the function of the graph and the logical data set used by the graph.

In Fig. 2, the FEED parameter is a design time parameter that affects the logical operation of the gather.mp subgraph. For example, the sort component 216 of the first subgraph 202 may sort the data it receives in ascending order, while for one value of the FEED parameter, another different value of the FEED parameter May cause the alignment component 216 to align the data in descending order. In some instances, a dataflow graph that includes design-time parameters is called a "generic graph" because its logic operation changes based on the supplied values of the design-time parameters.

The DATE parameter is a run-time parameter supplied on a job-by-job basis without affecting the logical operation of the subgraph 202. [

1.2 Parameter sets

In some instances, commonly used sets of parameters for dataflow graphs are stored as "parameter sets" (sometimes called "psets") that can be stored on disk and easily reused. For example, in FIG. 2, the first subgraph 202 has three psets, PSET_mexico 218, PSET_canada 220, and PSET_usa 222 associated with it. PSET_mexico 218 includes a commonly used DATE parameter value "today () ", which is a function that returns a commonly used FEED parameter value" mexico " PSET_canada 220 includes a commonly used FEED parameter value "canada" and a commonly used DATE parameter value "today ()". PSET_usa 222 includes a commonly used FEED parameter value "usa" and a commonly used DATE parameter value "today ()".

Similarly, the second subgraph 204 has a single pset, PSET 223, associated with it. PSET 223 has a commonly used DATE parameter value "today () ", which is a function that returns the current date.

2 parameter analysis module

In some instances, prior to the dataflow graph 117 being executed by the execution module 112, the parameter interpretation module 106 of FIG. 1 may be configured to generate the dataflow graph 117 (and its associated subgraphs 202, 204 )) And determines the number of unique design-time parameters in the one or more psets. For each unique design time parameter for a given datagram graph, the parameter interpretation module 106 instantiates a separate executable instance of the datagram graph. 3, three instances of the first subgraph 202 are instantiated (PSET_mexico-> gather.mp 202a) for the dataflow graph 217, gather.mp of FIG. 2, , PSET_canada-> gather.mp (202b), PSET_usa-> gather.mp (202c)), each instance is configured according to a different one of the three unique feed parameters of psets, mexico, canada, do. Since only a single instance of the second subgraph 204 (process.mp 204a) is instantiated at run time, since the second subgraph 204 is related only to a single pset 223 that does not include any design time parameters, do.

Once the appropriate instances of the subgraphs 202 and 204 are instantiated by the parameter interpretation module 106, the parameter interpretation module 106 may determine the parameters of the parameterized paths for the data sets to actual parameter values from psets Value placeholders to interpret the path to the physical location of the data sets. For example, for the PSET_mexico-> gather.mp instance 202a of the first subgraph 202, the path to the first data set 206 is the path with the FEED parameter value of 'mexico' and the DATE parameter value of '031014 It is interpreted as / mexico / inv_031014.

If the parameter interpretation module 106 instantiates the dataflow graph 217 including its subgraphs 202 and 204 and interprets the physical paths to the datasets of the dataflow graph 217, (217) is prepared for execution by the execution module (112). During execution, the three instances 202a, 202b, and 202c of the first subgraph 202 read data from their respective input data sets, process the data, and write /trans_031014.dat to the physical file And stores the data. Because the data set (e.g., DS4 212) for the instance 204a of the input second subgraph 202 is interpreted as the same physical file as the output data set of the first subgraph, the /trans_031014.dat The physical file is read by an instance of process.mp, processed and stored in table 214. [

3. Data lineage module

4, in some instances, rather than executing the dataflow graph 217, the data designer or business user 121 of FIG. 1 may check the lineage of the data as it passes through the dataflow graph 217 It may be necessary to do so. To that end, the data lineage module 115 of FIG. 1 is configured to analyze the dataflow graph 217 to generate a data lineage report for providing to the data designer or business user 121. [

In some instances, the data lineage module 115 identifies the individual subgraphs 202, 204 of the dataflow graph 217 as a first step in determining the data lineage for the dataflow graph 217. For each of the identified subgraphs 202, 204, the data lineage module 115 identifies one or more psets 218, 220, 222, 223 associated with the subgraph 202, 204, Determines the number of design time parameters unique to one or more psets (218, 220, 222, 223) for the subgraphs (202, 204). For each unique design-time parameter, the parameter interpretation module instantiates a separate instance of the subgraph 202, 204.

In some instances, the data lineage module 115 operates under the assumption that the actual physical files and the data they store are independent of the data lineage analysis. For this reason, any runtime parameter values used to interpret the physical location of the data sets are unnecessary and can be replaced with placeholder values. As described above, for each run-time parameter associated with the subgraph, the corresponding placeholder, correction analysis parameter value is included in the subgraph. For example, in FIG. 2, since both the dataflow graphs 202 and 204 include a DATE runtime parameter, they also all include the static analysis parameter value of the placeholder, 'MMDDYY'.

When the data lineage module 115 analyzes the dataflow graph 217 to determine the data lineage, all instances of the DATE parameter of the dataflow graph are replaced by the placeholder value 'MMDDYY' And creates temporary data set objects 452 as shown. Interconnections between the various subgraph instances and the temporary dataset objects are then identified and provided to the data designer or business user as a data lineage. For example, an analysis of the instances 202a, 202b, 202c of the first subgraph 202 may be performed by determining whether all instances of the first subgraph 202 are in the data set represented by the / trans_MMDDYY.dat data set object And recording the data. The analysis then shows that the instance 204a of the second dataflow graph 204 reads from the dataset represented by the dataset object / trans_MMDDYY.dat . Based on this information, the data lineage for the dataflow graph 217 is such that the output of the instances 202a, 202b, 202c of the first subgraph 202 is the output of the instances 204a of the second subgraph 204 Lt; / RTI >

4. How to discover and create a logical pset

In some instances, a given dataflow graph is executed using an execute command that receives the pyrameter values as arguments supplied to the execute command rather than from the previously stored pset. Since the above-described method uses only stored psets to determine the data lineage, the psets associated with the parameter values resulting from the arguments supplied to the execution command for execution of the dataflow graph are not displayed in the data lineage. This may result in an incomplete or inaccurate data lineage to the corporate designer or auditor.

5 illustrates a method for augmenting a repository of existing logical parameter sets (psets) for a dataflow graph with the logical psets generated based on the identified parameter sets in the log associated with the execution of instances of the dataflow graph Order also. In some instances, the method described in FIG. 5 is implemented by the data lineage module 115 of FIG.

4.1 Graph Parameters

One example of a data flow graph initially (e.g., the first subgraph 202 of FIG. 1) includes two parameters P1 and P2, each of which may be a "design time & "Parameter. As described above, the design-time parameter is a parameter that affects the logical operation of the graph (for example, it may change the transformation performed by the graph), while the run-time parameter is a task-specific ) And does not affect the logical operation of the graph.

4.2 Parameter classification

The graph 202 is provided to a parameter classification step 424 for analyzing the parameters of the graph 202 to produce a parameter classification result 426. [ In the parameter classification result 426, each parameter is classified as a design time parameter or a run time parameter. In the exemplary case described in the flowchart, P ₁ is classified as a design time parameter and P ₂ is classified as a run time parameter.

In some instances, the parameters for the design flow graph are pre-classified as design-time or run-time parameters (e.g., by the user). In another example (for legacy dataflow graphs), the parameters for the dataflow graph are not pre-classified as design-time or run-time parameters. In this case, the parameter classifying step 424 may assume that all parameters are design-time parameters. In a subsequent reclassification step, it is determined that a given parameter has a number of unique values (e.g., exceeding a predetermined threshold) in a collection of log entries (e.g., the job log data store described below) The given parameters can be reclassified as run-time parameters. Alternatively, reclassification may be based on data lineage sensitivity analysis. In particular, if the parameters can take on a variety of different values without changing the data lineage within the datagram graph (i. E. The influence or dependence of the data sets or components in the datagram graph), then the parameters are classified as runtime parameters . For example, if the associated record formats or other characteristics of a data set (e.g., DS1, DS2, DS3 in FIG. 3) in the graph are not affected by the various values of the parameter, do. This data lineage sensitivity analysis, such as more comprehensive data lineage sensitivity analysis, including interpreting all internal influences and dependencies, and more limited data lineage sensitivity analysis, including interpreting only the effects and dependencies associated with data set record formats May be used.

In some examples (e.g., for legacy dataflow graphs), the parameters may include both design time and runtime portions. For example, the filename parameter " /mexico/inv_031014.dat "may be a hybrid parameter in that it includes a design-time portion (i.e.," mexico ") and a run- . In this example, the user may use a regular expression of the string parsing rules used by the parameter classification step 424 to extract and classify each design time and runtime parameters from the hybrid parameter, Can be provided.

4.3 Activity log data store

The method utilizes an activity log data store 428, which includes a plurality of activity log entries 429, each containing information related to the execution of instances of the data flow graph 202. Among other information, at least some of the work entries include a record of the run command that was used to instantiate the dataflow graph 202. [ The execution command for a given job log entry includes parameter values and graph names that were supplied as arguments to the execution command. Generally, at least some of the job log entries of the job log data store 428 instantiate the dataflow graph without accessing any parameter sets, but instead receive parameter values with arguments supplied to the run command.

4.4 Processing Loop

The operation log data store 428 and the parameter classification result 426 generate a new logical pset for the graph execution command for each activity log entry 429 of the activity log data store 428, It is provided to the processing loop 430 to determine if the logical pset already exists in the store 448 of the existing logical psets and to add the new logical pset to the store 448 if it does not already exist.

4.4.1 Initial Command Line Logical pset Configuration

Within the processing loop 430, the parameter classification result 426 and the job log entry Jn 432 from the job log data store 428 are stored in the parameter classification result 426 to generate a logical pset 436 (Step 434) of analyzing the operation log entry 432 in accordance with the operation log entry 432 of FIG. To do so, the logical pset configuration step 434 analyzes the graph execution command contained in the job log entry 432 to extract the parameter values included as arguments to the graph execution command. The logical pset configuration step 434 also extracts a project scope contained in the job log entry 432. [ In some examples, the project scope is configured to display the project in which the dataflow graph is running, an indication of internal parameters for the dataflow graph, and preferences used by the dataflow graph, a representation of global variables and configuration variables .

The logical pset configuration step 434 automatically includes the extracted project scope in the logical pset 436. The logical pset configuration step 434 then matches each extracted parameter value with the corresponding parameter in the parameter classification result 426. [ If the logical pset configuration step 434 determines that the extracted parameter value corresponds to the design time parameter of the parameter classification result 426, then the logical pset configuration step 434 determines whether the design extracted in the logical pset 436 Contains the value of the parameter. If the logical pset configuration step 434 determines that the extracted parameter value corresponds to a run time parameter of the parameter classification result 426, then the extracted parameter value is not included in the logical pset 436.

4.4.2 Calculating the pset signature string

The logical pset 436 is provided to the pset signature string computation step 442 which computes the logical pset signature string 444 based on the parameter values of the logical pset 436 and the project scope. In some examples, the pset signature string 444 includes a project scope for the logical pset 436, name / value pairs of parameters of the logical pset 436, Prototypes are serialized by serialization. In another example, the pset signature string 444 is computed by applying a hash function or other data mapping algorithm to the logical pset 436.

4.4.3 Searching for pset signature strings

The pset signature string 444 is provided to the pset signature search step 446 along with the pset signature strings of all existing logical psets in the store 448 of the existing logical psets. For each of the existing logical psets, the pset signature string of the existing logical pset is compared to the pset signature string 444. If the pset signature string 444 matches at least one of the pset signature strings of existing logical psets then the logical pset for the execution command instantiation of the graph 432 already exists in the store 448 of the existing logical psets So nothing needs to be done.

In some examples, the pset signature string of all existing logical psets in the store 448 of all existing logical psets is stored in the store 448 along with the existing logical psets. In another example, the signature string for the existing logical psets is calculated on the fly and as needed.

4.4.4. Adding a new logical pset

Otherwise, if none of the signature strings of the existing logical psets match the pset signature string 444, then the logical pset 436 and its signature string 444 are updated by a new logical pset addition step 450 Is added as a new logical pset to the store 448 of the logical psets.

4.4 Yes

Referring to Figs. 6 and 7, an exemplary operation of the logical pset discovery and generation method of Fig. 4 as applied to the first sub-graph 202 of Fig. 2 is presented. The first subgraph 202 of FIG. 2 includes two parameters, P1 = FEED and P2 = DATE. The first subgraph 202 is provided to a parameter classification step 424 where parameters are classified as "design time" or "run time" parameters to produce a parameter classification result 426. The parameter classification result 426 indicates that the P1 (FEED) parameter is a design-time parameter and the P2 (DATE) parameter is a run-time parameter.

The parameter classification result 426 and the operation log data store 428 are provided to a logical pset configuration step 434. [ In the example of FIG. 6, the job log data store 428 includes four job log entries that contain information related to the execution of instances of the first subgraph 202 (i.e., gather.mp). Each job log entry includes an execution command that received values for DATE and FEED parameters as arguments.

The logical pset configuration step 434 generates a different logical pset 436 for each of the job log entries of the job log data store 428. [ Because the P1 (FEED) parameter is a design time parameter, its value (e.g., mexico, usa, canada, or hong kong) that was supplied as an argument to the execution command is included for each of the logical psets 436. Because the P2 (DATE) parameter is a run-time parameter, its value that was supplied as an argument to the execute command is not included in the logical psets 436. Each of the logical psets 436 includes a project scope for a corresponding instance of the first subgraph 202.

7, the logical psets 436 are provided to the pset signature string computation step 442, which computes different logical pset signature strings 444 for each of the logical psets 436.

The set of logical pset signature strings 444 and the set of logical pset signature strings for existing psets 447 of the existing psets repository 448 are provided to the retrieval step 446. [ As in the case of FIG. 2, there are three conventional psets associated with the first subgraph 202, one for the mexico FEED parameter, one for the usa FEED parameter, and one for the canada FEED parameter. Thus, the set of logical pset signature strings 444 for the existing psets 447 includes a string for each of the existing psets associated with the first subgraph 202.

The retrieval step 446 retrieves the presence of each of the logical pset signature strings 444 in the set of logical pset signature strings for the existing psets 447. [ In this example, the result generated by the retrieval step 446 is a logic having a unique logical pset signature string not included in the set of logical pset signature strings for the existing psets 447 having the FEED parameter value of 'hongkong' It is a logical pset signature string associated with pset.

 the logical pset 436 including the 'hongkong' feed parameter and the logical pset whose result of the search step 446 includes the FEED parameter of 'hongkong', and its corresponding logical pset signature string 444, To the store 448 of the logical pset (step 450).

By adding a new logical pset to the repository, a 'hongkong' instance of the first subgraph 202, which may have been overlooked in previous data lineage results, will be indicated in the data lineage results.

While static analysis values for runtime parameters are described as being stored in the dataflow graph itself in the examples above, in some instances, static analysis values for runtime parameters are maintained in one or more psets associated with the dataflow graph .

In some instances, certain design-time parameter values are derived (e.g., from a database) from sources that are not necessarily present at the static analysis time. However, in some examples, the job log entries stored in the job log data store include values for all parameters that were interpreted for that particular job. At the static analysis time, the stored values may be used in place of the parameter values derived from sources that do not exist at the static analysis time.

In some examples, the job log entries of the job log store include all analyzed parameters for the dataflow graph, a log of all files read and written by the dataflow graph, and performance tracking information. In some instances, the job log entries of the job log data store are augmented with any set of logical parameters found by the method of FIG. In some instances, augmenting the job log entries of the job log data store with found logical parameter sets includes forming an association between the job log entries and the found logical parameter sets. The enhanced job log entries of the job log data store can be utilized to provide various types of information to the data designer or business user. In some instances, the enhanced job log entries may be analyzed to ensure that logically linked dataflow graphs are also physically connected. In some instances, the enhanced job log entries may be analyzed to determine which physical data set corresponds to which logical data set instances. In some instances, the enhanced job log entries may be analyzed to identify data sets that have the same physical filename but are associated with different static analysis parameters. In this example, an inconsistency may be presented to the user for manual recovery or an inconsistency may be automatically recovered. In some instances, the data lineage report may include an indication of an inconsistency and whether it can be automatically recovered.

In some instances, the enhanced job log entries may be used by the data lineage module to filter data lineage reports by frequency and / or recency. For example, the metadata environment module may maintain multiple dataflow graphs and psets that are no longer executed by the execution module. These dataflow graphs and psets can be left in place only when needed later. However, non-executing dataflow graphs and psets can cause unnecessary confusion in data lineage reports. To reduce this confusion, the enhanced job log entries may be analyzed to determine which dataflow graphs and / or psets are occasionally used and / or have not been used recently. Based on this frequency and up-to-date information, data flow graphs and psets (for example, dataflow graphs that have not been run for the past year) that have not been executed occasionally and recently have been presented to enterprise users, Lt; / RTI >

In some instances, there may be a logical pset for a given dataflow graph (e.g., a pset containing FEED = USA), but one or more operations that call the dataflow graph may not directly utilize the existing pset Values to the dataflow graph. In this case, the association maintained between the tasks and the logical psets accessed by the tasks (e.g., through the signatures associated with the tasks) may be used to group the task log entries based on the associated logical psets . Based on the grouping, any tasks that are instantiated by calling a graph directly instead of utilizing an existing pset can be identified as being related to the logical pset and its parameters.

In some examples, each job log entry for a dataflow graph includes a list of all interpreted parameter values for execution of a dataflow graph associated with the job log entry among other information. Once a number of job log entries have been accumulated, the interpreted parameter values contained in the job log entries may be compared to identify various "design time instances" of the datagram graph. For example, certain parsed parameters in the job log entries may be represented by only a few values in all of the job log entries, while certain other interpreted parameters may be represented in many of the job log entries Lt; / RTI > These interpreted parameters, which are represented by only a few values in the job log entries, are likely to be "design time" parameters, and other interpreted parameters, represented by many different values in the job log entries, There is a possibility. Any instances of a dataflow graph sharing a unique combination of "design time parameters" are grouped together and are all considered a "design time instance" of the dataflow graph. The data lineage module includes different design-time instances of the datagram graph in the data lineage report.

5. How to discover and mitigate redundant logical data sets

5.1 Overview

In general, input and output data sets (e.g., databases or tables of data) for a given dataflow graph are specified as logical data sets in the dataflow graph. In some examples, each logical data set is associated with an identifier, such as a logical file name.

And interpreting each logical data set to a corresponding set of physical data (e.g., files on disk) before the dataflow graph is executed.

In some instances, each physical data set is associated with an identifier, such as a physical file name (e.g., "summary.dat"). The parameter resolution process can successfully resolve the logical data set to its corresponding physical data set even if the logical file name of the logical data set is different from the physical file name of the corresponding physical data set.

When a data lineage report is determined for a dataflow graph comprising two or more subgraphs, the lineage relationships between the subgraphs are determined at least in part according to logical file names of the input and output logical data sets of the two or more subgraphs . For this reason, the accuracy of the lineage relationship requires that any input and output logical data sets of the two or more subgraphs that reference a given physical data set share the same logical file name. In practice, a first subgraph writes to a given physical data set and a second subgraph thereafter reads from the given physical data set, but the output logical data set of the first subgraph and the input logical data If the logical file names in the set are not matched, no lineage relationship will be identified between the two subgraphs. In some examples, two logical data sets having logical file names that are interpreted as the same physical data set but not matched are referred to as "duplicate logical data sets ".

As will be described in greater detail below, redundant logical data sets can be identified and presented to the user in the dataflow graph. The user may then choose to process the redundant logical data sets in various ways.

5.2 Example without redundant logical data sets

8, an example 817 of a data flow graph generated using the development environment 118 of FIG. 1 includes a first subgraph 802 named gather.mp and a second subgraph 802 named precess.mp 2 < / RTI >

The first sub-graph 802 is input to a second logic data set D _L2 (808) having a first logic data set D _L1 (806) and the logical file name "Acct_2.dat" has a logical file name "Acct_1.dat" . The first subgraph 802 processes data from the first and second logical data sets 806 and 808 and writes the third logical data set D _L3 810 having the logical file name " Acct_summ.dat & And records the result of the processing. The second subgraph 804 receives as input the third logical data set D _L3 810 with the logical file name " Acct_summ.dat & quot ;, processes the data from the third logical data set 810 , And records the processing result in the table 814. Note that the third logical data set 810 used by both the first subgraph 802 and the second subgraph 804 has the same logical file name in both subgraphs 802 and 804 .

Referring to FIG. 9, when the dataflow graph 817 is interpreted prior to execution, the logical data sets are interpreted as their corresponding sets of physical data. For example, if the first logical data set 806 is interpreted as the first physical data set D _P1 814 having the physical file name " Acct_1.dat & quot ;, and the second logical data set 808 is interpreted as the physical file name " Acct_2. dat is interpreted as a third physical data set D _P3 (818) having an "interpreted as a second physical data set D _P2 (816) and having a third logic data set (810) a physical file name" summary.dat ".

Referring to FIG. 10, a data lineage report 1017 for the data flow graph includes a first subgraph 1002, a second subgraph 1004, a first logical data set 1006, 1008, and a third logical data set 1010. The data lineage report 1017 also includes a first lineage relationship 1018 between the first logical data set 1006 and the input of the first subgraph 1002, 1 subgraph 1002 and a third genealogical relationship 1022 between the output of the first subgraph 1002 and the third logical data set 1010 and a second genealogical relationship 1020 between the inputs of the first subgraph 1002 and the third 3 logical data set 1010 and the second subgraph 1004, as shown in FIG. (I.e., the third logical data set D _L3 810) having the same logical file name (i.e., " Acct_summ.dat ") is output to the output of the first subgraph 802, Note that the data lineage report 1017 is correct in this case because it exists at the input of the graph 804.

5.3 Example with redundant logical data sets

11, another example 1117 of a dataflow graph generated using the development environment 118 of FIG. 1 includes a first subgraph 1102 named gather.mp and a second subgraph 1102 named process.mp 2 < / RTI >

The first subgraph 1102 inputs a first logical data set D _L1 1106 having a logical file name " Acct_1.dat " and a second logical data set D _L2 1108 having a logical file name " Acct_2.dat " . The first subgraph 1102 processes the data from the first and second logical data sets 1106 and 1108 and outputs the result of the processing to a third logical data set D _L3 having a logical file name " Acct_summ.dat & (1110). The second subgraph 1104 receives as input a fourth logical data set D _L4 1111 having a logical file name " Acct-summ.dat " and processes the data from the fourth logical data set 1111 And records the processing result in the table 814. [ The logical file name for the third logical data set 1110 (i.e., " Acct_summ.dat ") is different from the logical file name for the fourth logical data set 1111 (ie, " Acct-summ.dat ") Please note.

Referring to FIG. 12, when the data flow graph 1117 is interpreted prior to execution, the logical data sets are interpreted as their corresponding sets of physical data. For example, the first logical data set 1106 is interpreted as a first physical data set D _P1 1114 having a physical file name " Acct_1.dat & quot ;, and the second logical data set 1108 is interpreted as a physical file name & Acct_2.dat ", and the third logical data set 1110 and the fourth logical data set 1111 are both interpreted as a second physical data set D _P2 1116 having a physical file name" summary.dat " Is interpreted as a third physical data set D _P3 (1218). The third logical data set 1110 and the fourth logical data set 1111 are referred to as redundant logical data sets because they each indicate the same physical data set (i.e., the third physical data set 1218) .

13, a data lineage report 1317 for the data flow graph includes a first subgraph 1102, a second subgraph 1104, a second logical data set 1106, a second logical data set 1108, a third logical data set 1110, and a fourth logical data set 1111. The data lineage report 1317 also includes a first lineage relationship 1318 between the first logical data set 1106 and the input of the first subgraph 1102, A second genealogical relationship 1320 between inputs of the first subgraph 1102, a third genealogical relationship 1322 between the first subgraph 1102 and the third logical data set 1110, 4 logical relationship between the 4 logical data set 1111 and the second subgraph 1104.

Two different sets of logical data (i.e., the third logical data set 1110 and the fourth logical data set 1111) having different logical file names are stored in the same physical data set (i.e., the third physical data set 1218), it is noted that the data lineage report 1317 is inaccurate in this case. In particular, the third logical data set D _L3 (1110) having the logical file name "Acct_summ.dat" is present at the output of the first sub-graph 1102 and the file name having the logical "Acct-summ.dat" A fourth logical data set 1111 is present at the input of the second subgraph 1104. The data lineage report 1317 shows the third logical data set 1110 and the fourth logical data set 1111 as a separate data set having no lineage relationship with each other. Thus, the data lineage report 1317 incorrectly includes a break in the data lineage between the third logical data set 1110 and the fourth logical data set 1111.

5.4 Discovery of redundant logical data sets

In some instances, redundant logical data sets in the dataflow graph can be found by analyzing runtime artifacts (e.g., job logs 429 in FIG. 5) generated by execution of the dataflow graph have. Specifically, each time a dataflow graph is run, a job log is generated.

The activity log includes information related to the execution of the dataflow graph including the graph instance name and the access type (read or write) for each dataset component of the graph it accessed. Graph instances may be examined to determine logical dataset names for each dataset component. By matching the graph instance with the data component name, the system can map logical data set names to physical data set names.

To identify redundant logical data sets, the operational logs are analyzed to identify any logical to physical dataset mappings that are different from the second logical data set of the first logical data set of the mapping. Any logical to physical data set mappings that differ from the first logical data set and the second logical data set are classified into redundant logical data sets.

The identified redundant logical data sets are presented or automatically mitigated to the user who determines to correct the redundant logical data sets.

5.4.1 Example of Discovery of Redundant Logical Data Sets

Referring back to FIG. 12, when the interpreted dataflow graph 1117 is executed, a job log for the dataflow graph execution is generated.

The job log includes a single logical to physical data set mapping corresponding to the flow between the first subgraph 1102 and the second subgraph 1104. The logical to physical data set mapping comprises an identifier for the third logical data set D _L3 1110 at the output of the first subgraph 1104 and an identifier for the fourth logical data set D 1110 at the input of the second subgraph 1106. [ An identifier for _L4 1111, and an identifier for the third set of physical data 1218.

The third logical data set 1110 and the fourth logical data set 1111 are stored in separate logical data sets (e.g., different logical file names) that point to the same physical data set (i.e., the third physical data set 1218) The third logical data set 1110 and the fourth logical data set 1111 are classified into redundant logical data sets.

While the above described simple example includes the identification of a single pair of redundant logical data sets from a single operation log in an actual implementation of the data processing system including the redundant logical data set discovery method, And < / RTI >

5.5 Reduced redundant logical data sets

As noted above, redundant logical data sets may result in a break in data lineage reports. Once redundant logical data sets are identified, several different methods can be taken to remove redundant logical data sets or mitigate its impact on data lineage reports. In some instances, the identified redundant logical data sets are presented to the user, for example, in the form of a spreadsheet. The user may then edit the dataflow graphs containing redundant logical data sets to remove redundant logical data sets (e.g., by ensuring that a given physical data set is referenced only by a single logical data set in a given dataflow graph) . In another example, a user may mark pairs of redundant logical data sets as equivalent. In this way, the user does not need to make any changes to the dataflow graphs. In another example, a pair of redundant logical data sets may be automatically marked as equivalent.

When pairs of redundant logical data sets are marked as equal, there are several ways to display equivalence in the data lineage report. In one method, a set of physical data referred to by a pair of redundant data sets is shown connected to redundant logical data sets in a data lineage report. For example, referring to FIG. 14, a third physical data set D _P3 1218 is included in the data lineage report 1317. Both the third logical data set D _L3 1110 and the fourth logical data set D _L4 1111 are shown connected to the third physical data set 1218 by the lineage relationships 1450 and 1452. [

In yet another method, logical data sets of a pair of redundant logical data sets are shown connected together in a data lineage report by a lineage relationship. For example, referring to FIG. 15, it is seen that in the data lineage report 1317, a third logical data set D _L3 1110 is connected to a fourth logical data set D _L4 1111 by a lineage relationship 1550 .

In another method, a pair of redundant logical data sets is represented by a combined logical data set in a data lineage report. For example, referring to FIG. 16, a pair of redundant logical data sets is represented by a combined logical data set D _LR 1654 in a data lineage report 1317.

In another method, one logical data set of pairs of redundant logical data sets is selected to represent a pair of redundant logical data sets in a data lineage report. For example, referring to FIG. 17, a fourth logical data set D _L4 1111 represents a pair of redundant logical data sets in a data lineage report 1317.

In another method, a representation of a combined logical data set of a pair of redundant logical data sets and a pair of redundant logical data sets is included in the data lineage report. A unique configuration of lineage relationships between a pair of redundant logical data sets and a combined logical data set is shown in the data lineage graph. 18, the data lineage report 1317 includes a combined logical data set representation D _LR 1854, a third logical data set D _L3 1110, and a fourth logical _< RTI ID = 0.0 > Data set D _L4 1111. The combined logical data set 1854 is shown to have direct lineage relationships with the first subgraph 1102 and the second subgraph 1104. The combined logical data set 1845 also has an indirect genealogical relationship with the first subgraph 1102 via the third logical data set 1110 and a second subgraph (1104). ≪ / RTI >

In another method, logical data sets of a pair of redundant logical data sets are included in the data lineage report. A unique configuration of lineage relationships between logical data sets of pairs of redundant logical data sets is shown in the data lineage graph. For example, referring to FIG. 19, a data lineage report 1317 includes a third logical data set D _L3 1110 and a fourth logical data set D _L4 1111. The fourth logical data set 1111 is shown to have direct lineage relationships with the first subgraph 1102 and the second subgraph 1104. The third logical data set D _L3 1110 has a direct descendent relationship with the first subgraph 1102 and with the second subgraph 1104 through the fourth logical data set 1111 .

In some instances, it is noted that the mitigation methods described above are represented in the data lineage reports as dashed lines, bold lines, or in some other alternative manner to clarify to the user of the data lineage report that the mitigation method has been applied to the data lineage report.

While the redundant logical data set discovery and mitigation approaches are described using scenarios in which a first component writes to a physical data set and another component reads from the physical data set, other scenarios may result in redundant logical data sets have. For example, a pair of redundant logical data sets may be the result from two different logical data sets that are read from the same physical data set. Similarly, a pair of redundant logical data sets may be a result from two different logical data sets writing to the same physical data set.

The foregoing methods are described in U.S. Serial No. 12 / 393,765, filed February 26, 2009, U.S. Serial No. 13 / 281,039, filed October 25, 2011, To incorporate features from various other methods for managing and presenting data lineage information and for managing data objects as described in more detail in U. S. Provisional Application No. 62 / 028,485,

The methods described above may be implemented using, for example, a programmable computing system that executes appropriate software instructions, or may be implemented in suitable hardware, such as a field programmable gate array (FPGA) or in some hybrid form. For example, in a programmed manner, the software may be one or more programmed or programmable computing systems (which may be a variety of architectures such as distributed, client / server, or grid), each of which may be implemented as at least one processor (volatile and / At least one data storage system (including at least one input device or port) for receiving an input using at least one input device or port and at least one output device or port for providing output ) Comprising at least one user interface. The software may include, for example, one or more modules of a larger program that provide services related to the design, construction and execution of a dataflow graph. The modules of the program (e.g., elements of the dataflow graph) may be implemented with data structures or other organized data that conforms to a data model stored in a data repository.

The software may be provided on a non-volatile medium, such as a CD-ROM or other type of computer readable medium (e.g., readable by a general purpose or special purpose computing system or device) Type, non-transitory medium (e. G., Encoded in a radio signal) over a communication medium of the network. Some or all of the above processes may be performed on special purpose computers or using special purpose hardware such as coprocessors or field programmable gate arrays (FPGAs) or dedicated application specific integrated circuits (ASICs). The processing may be implemented in a distributed manner in which different portions of the computation specified by the software are performed by different computing elements. Each such computer program is stored on a computer of a storage device, preferably a general purpose or special purpose computer accessible by a general purpose or special purpose programmable computer, for configuring and operating the computer when the storage device medium is read by a computer performing the processing described herein Readable storage medium (e.g., solid state memory or medium, or magnetic or optical medium)). The system of the present invention may also be considered to be implemented as a type of non-transitory medium, which is configured as a computer program, and such configured medium may be a medium in which a computer is programmed to perform a specific and predefined .

A number of embodiments of the present invention have been described. Nevertheless, the foregoing description is for the purpose of illustration and is not intended to limit the scope of the invention, which is defined by the scope of the following claims. Accordingly, other embodiments are also within the scope of the following claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be order-independent and thus may be performed in a different order than described.

Claims (17)

A method for managing lineage information, the method comprising:
Receiving lineage information representative of at least one lineage relationships between at least two data processing programs and at least two logical data sets;
Receiving at least one runtime artifacts, each runtime artifact comprising information relating to a previous execution of a data processing program of the at least two data processing programs; And
And analyzing the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information. The method according to claim 1,
Wherein the at least one candidate modifications comprise a candidate modification to add a new indirect genealogical relationship between a data processing program of the at least two data processing programs and a logical data set of the at least two logical data sets, Way. The method according to claim 1,
Wherein the at least one candidate modifications comprise a first candidate modification that adds a new direct lineage relationship between a data processing program of the at least two data processing programs and a logical data set of the at least two logical data sets. How to manage. The method of claim 3,
Wherein analyzing the runtime artifact and the genealogy information comprises analyzing logs of previous runs of the at least two data processing programs to determine physical data sets to be read or written by the at least two data processing programs A method for managing genealogical information, comprising: 5. The method of claim 4,
Wherein analyzing the runtime artifact and the genealogy information further comprises identifying two separate sets of logical data of the at least two logical data sets represented by the genealogy information and associated with the same set of physical data. How to manage. 6. The method of claim 5,
Wherein the first candidate modification comprises creating a new lineage relationship between the two separate logical data sets. 6. The method of claim 5,
Wherein the first candidate modification includes merging the two separate sets of logical data into a new combined logical data set. The method according to claim 1,
Wherein each data processing program of the at least two data processing programs is an instance of a generic data processing program instantiated according to a set of at least one parameter values. 9. The method of claim 8,
The step of analyzing the at least one runtime artifact and the genealogy information
At least one log of previous executions of the first data processing program of the at least two data processing programs to a first set of parameter values, the first parameter being used in a first instantiation of the first data processing program To determine the set, analyze,
Selecting at least some parameters from the first set of parameters, and
And determining that the first instantiation of the first data processing program is not represented in the genealogical information based on the generic version of the first data processing program and the at least some parameters. Lt; / RTI > 10. The method of claim 9,
Wherein selecting at least some parameters from the first set of parameters comprises selecting parameters based on information received from a user. 10. The method of claim 9,
Wherein selecting at least some parameters from the first set of parameters comprises selecting parameters based on at least one predefined rule. 12. The method of claim 11,
Wherein the first rule of the at least one predefined rule specifies that parameters having parameter values in date form are excluded from the selected parameters. 12. The method of claim 11,
Wherein the first rule of the at least one predefined rule specifies that a parameter having a parameter value converted into logic of the generic data processing program is included in the selected parameters. 10. The method of claim 9,
Wherein the at least one candidate modifications to the genealogy information include a first candidate modification that adds a new genealogical relationship between the first data processing program of the at least two data processing programs and the logical data set of the at least two logical data sets Wherein the method comprises the steps of: 22. A software for managing lineage information stored in a non-transitory form on a computer readable medium, the software comprising:
Receive lineage information representative of at least one lineage relationships between at least two data processing programs and at least two logical data sets;
To receive at least one runtime artifacts, each runtime artifact comprising information relating to a previous run of a data processing program of the at least two data processing programs; And
And to analyze the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information. 1. A computing system for managing lineage information, the system comprising:
An input device or port configured to receive lineage information and at least one runtime artifacts indicative of at least one lineage relationships between at least two data processing programs and at least two logical data sets, each runtime artifact comprising: Comprising information relating to a previous execution of a data processing program of at least two data processing programs; And
And at least one processor configured to analyze the genealogy information and the at least one runtime artifacts to determine at least one candidate modifications to the genealogy information, Computing system for. 1. A computing system for managing lineage information, the system comprising:
Means for receiving lineage information and at least one runtime artifacts representing at least one lineage relationships between at least two data processing programs and at least two logical data sets, each runtime artifact comprising: Comprising information relating to previous execution of a data processing program of the data processing programs; And
And means for analyzing the at least one runtime artifacts and the genealogy information to determine at least one candidate modifications to the genealogy information. .

Images