JP4846736B2 - Parallel processing support device - Google Patents

Description

  The present invention relates to a technique for processing a large amount of data files such as large-scale simulation data in the field of computer science.

  The result of a simulation performed in a large-scale simulation system such as an ocean simulation system is composed of a large amount of data files. It is not uncommon for a large number of data files to be tens of terabytes or more in size and 10,000 or more files. In general, such a large amount of data files are not created in the same format and often have slightly different formats depending on the contents of the data.

  In numerical simulation, the intersection of line segments as shown in FIG. 16A is generally defined as a calculation grid, and processing (for example, calculation of physical quantities) on numerical data proceeds based on this calculation grid. However, due to the physical characteristics of the numerical data and various circumstances such as the convenience of calculation formulas, processing (calculation of physical quantities) for some numerical data is performed based on different calculation grids as shown in FIG. 16B. There is often.

  Here, the first data file calculated based on the calculation grid shown in FIG. 16A (referred to as “first calculation grid”) and the calculation grid shown in FIG. 16B (“second calculation grid”). Suppose that there is a second data file calculated based on “. Furthermore, it is assumed that data existing in an area (cutout range) from 120.1 degrees to 121.6 degrees east longitude is cut out from the first and second data files.

  In this case, data corresponding to two calculation grids are cut out from the first data file, and data corresponding to four calculation grids are cut out from the second data file. However, when it is desired to cut out data corresponding to the four calculation grids from the first data file, the data cut out in accordance with the above-described cutting range (cutout data) is insufficient.

  Data indicating the details of the data in the data file, such as what kind of calculation grid the first data file includes, is referred to as “metadata” in this specification. .

  Conventionally, metadata is stored separately from simulation data, or a user designates and inputs corresponding metadata for each simulation data file to be processed.

  When the simulation data generated as a result of a large-scale simulation is used as a data processing specification after simulation, it is a great effort for the user to specify and input metadata for each of these large data files. Will be forced to pay. There is also a risk that the user may specify the metadata by mistake.

  In addition, parallel processing (parallel computation) is effective for efficiently performing a large amount of data processing, but it is necessary to prepare a control program (script) for causing the parallel computer group to execute the parallel computation. Traditionally, scripts have been written by users. For this reason, the user is required to have script description knowledge related to parallel computation, which impedes the ease of using the system and imposes a burden on the user. In addition, there is a possibility that proper parallel calculation may not be executed due to a script description error.

  An object of the present invention is to provide a technique capable of simply processing a large amount of data files.

  In order to achieve the above object, the present invention employs the following means.

That is, the present invention provides a parallel processing including a processing target data file group, the number of a plurality of calculation nodes in a parallel computer group performing parallel processing on the processing target data file group, and designation of processing contents for the processing target data file group. A receiving means for receiving the specified information;
Storage means storing usage and load status for each of a plurality of computing nodes included in the parallel computer group;
Based on the specified number of calculation nodes and the usage and load status, the specified number of calculation nodes performing parallel processing , and the processing targets for the specified number of calculation nodes Determining means for determining the arrangement of each processing target data file constituting the data file group;
A data placement command for placing each processing object data file on the designated number of calculation nodes according to the placement determination result by the determining means, and the processing object for each computation node on which each processing object data file is placed Control program generation means for automatically generating a parallel processing job script including a parallel processing execution statement of the data file group;
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing A file generation means for automatically generating a setting file including a file identifier of a target data file, an identifier of a calculation node in which the processing data file is arranged, and a description of a specified processing content;
Is a parallel processing support device.

Preferably, in the present invention, the determination unit is configured to process, for each processing target data file constituting the processing target data file group, processing target data from the specified number of calculation nodes included in the parallel computer group. Select a computation node having a storage capacity capable of storing a file and a processing result file for the file,
Among the selected computation nodes, the computation node having the smallest current processing load is determined as the computation node where the processing target data file is to be placed.

Preferably, in the present invention, the parallel processing designation information includes designation of a storage location of a processing result file generated as a result of processing on the processing target data file,
The control program generation unit generates the control program including a command statement indicating that the processing result file is transferred to the storage location.

Preferably, the present invention specifies the processing target data file group, the number of a plurality of calculation nodes in the parallel computer group that performs parallel processing on the processing target data file group, and the processing contents for the processing target data file group. Receiving means for receiving parallel processing designation information including;
Storage means storing usage and load status for each of a plurality of computing nodes included in the parallel computer group;
Determination means for deciding the arrangement of each processing target data file constituting the processing target data file group with respect to the specified number of calculation nodes based on the specified number of calculation nodes and the usage and load status When,
A data placement command for placing each processing object data file on the designated number of calculation nodes according to the placement determination result by the determining means, and the processing object for each computation node on which each processing object data file is placed Control program generation means for automatically generating a control program including a statement for executing parallel processing of data files, and
A parallel processing setting file that is referred to when each computation node in which each processing object data file is arranged performs processing of the processing object data file arranged in itself. A file generation means for automatically generating a configuration file including a file identifier of the data file, an identifier of a calculation node in which the processing data file is arranged, and a description of the specified processing content;
The data file to be processed is specified by selecting a file identifier selected from a list that displays multiple file identifiers of the data file that contains the file path of the data file stored in one of the multiple directories that make up the directory structure. Made by
Metadata storage means for storing metadata of processing target data;
A keyword list having keywords associated with the metadata;
The file path portion of the specified file identifier is used to extract a keyword for searching metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier. An extraction means for comparing a character string forming a part of the keyword list with the keyword list, and extracting a character string that matches at least one keyword in the keyword list as a keyword;
Search means for searching for metadata corresponding to the extracted keyword from the metadata storage means;
For each processing data file, in order to determine whether there is a related data file related to the processing data file, determine whether there is a related data file based on the metadata searched by the search means And a determination means,
When a processing target data file having a related data file is detected by the determination unit, the determination unit arranges the processing target data file and the related data file for the processing target data file in the same calculation node, and the control program generation unit Generating the control program for the processing target data file group including the related data file as one of the processing target data files, and generating the setting file for the related data file. You may do it.

  In addition, the present invention can be specified as a parallel processing support method and program having the same characteristics as the parallel processing support device described above, and a recording medium on which the program is recorded.

  According to the present invention, it is possible to easily process a large amount of data files. Further, according to the present invention, it is not necessary for the user to specify metadata for the processing target data.

It is a figure which shows the structural example of the simulation system which can apply this invention. It is a figure which shows the structural example of the computer for control shown in FIG. It is a figure which shows the structural example of the node shown in FIG. It is a figure which shows the example of a directory structure of the file database which stores the process target data file shown in FIG. FIG. 3 is a diagram illustrating an example of a data structure of a metadata table illustrated in FIG. 2. It is a figure which shows the example of a data structure of the use and load distribution condition table shown in FIG. It is a figure which shows the example of a display of the user interface (designation screen) provided to the user of a system. It is a figure which shows the example of a description of the file of the parallel processing designation | designated information input using a user interface. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the main routine of the creation process of the job script for parallel processing, and the setting file for parallel processing programs. It is a flowchart which shows the subroutine which concerns on analysis and acquisition of metadata. It is a flowchart which shows the subroutine of the search of the node used as the arrangement destination of a process target data file, and a determination process. It is a figure which shows the example of a description of the setting file for parallel processing programs. It is a flowchart which shows the execution process of a parallel processing program. It is a figure which shows the example of the calculation grid prepared as metadata with respect to process target data. It is a figure which shows the example of the calculation grid different from the calculation grid of FIG. 16A prepared as metadata with respect to process target data.

Explanation of symbols

X ... parallel computer group Y ... control computer 1, 11 ... CPU
2, 12 ... Main memory 3, 14 ... External storage device 7 ... Input device 8 ... Display device 6, 15 ... Communication interface 31 ... File database 32 ... Metadata table 33 ... Usage and load balancing information table

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configuration in the embodiment is an exemplification, and the present invention is not limited to the configuration in the embodiment.

[Simulation system]
FIG. 1 is a diagram showing a configuration example of a simulation system to which the present invention can be applied. In the example shown in FIG. 1, the simulation system includes a parallel computer group X and a control computer (information processing device) Y connected to the parallel computer group X via a communication line (network).

  The parallel computer group X is composed of a plurality of calculation nodes (nodes) # 0 to #n (n is a natural number) that performs parallel processing on a large number of data files constituting large-scale simulation data such as an ocean general circulation model.

  The computer Y manages simulation data (processing target data) to be processed in the parallel computer group X, and causes the parallel computer group X to execute parallel processing using the simulation data in accordance with a user operation. Control.

  The user of the simulation system provides parallel processing designation information for executing parallel processing of a large amount of processing target data (processing target data group) using the parallel computer group X through a UI (user interface) provided by the computer Y. input.

  Here, the parallel processing designation information includes a plurality of simulation data files (processing target data file group) to be processed in parallel processing, processing details of the parallel computer group Y for the processing target data file group (processing type, processing detailed parameters), It can include designation of a plurality of nodes (number of nodes) performing parallel processing, a storage position of a file (processed data file (processing result file)) generated as a result of parallel processing, and the like.

  The computer Y uses a parallel processing job script (a control program for the parallel computer group Y: hereinafter referred to as “script”) to give a parallel computer group X a control instruction related to parallel processing based on the input parallel processing designation information. And a parallel processing program configuration file (hereinafter also referred to as a “configuration file”) that is referenced when each node that executes parallel processing processes the data file to be processed. Generate automatically.

  In the script generation process, the computer Y acquires metadata (detailed information of the processing target data) for each processing target data file and determines the arrangement of the processing target data file group for the parallel computer group X. The metadata and the arrangement determination result are reflected in the description content of the script.

  The computer Y distributes (distributes) the processing target data file group to a plurality of nodes through execution of the script, and instructs the nodes to execute a parallel processing program (job). Each node executes the parallel processing program according to the description of the setting file, and performs processing on the distributed processing target data file based on the corresponding metadata. A processing result file is created through this processing. The processing result file is stored at a storage location specified as parallel processing specification information.

<Computer Y>
FIG. 2 is a diagram illustrating a configuration example of the computer Y. In FIG. 2, a computer Y includes a CPU 1, a main memory (MM: for example, RAM) 2, an external storage device (for example, a hard disk) 3, and input / output interfaces (I / F) 4 and 5 that are connected to each other via a bus B. , And a communication interface 6.

  An input device (keyboard, pointing device (eg, mouse)) is connected to the I / F 4, and a display device (display) 8 is connected to the I / F 5. . Further, the communication I / F 6 is connected to each of the nodes # 0 to #n via a communication line (network).

  The external storage device 3 includes a file database (file DB) 31 that stores a large amount of simulation data files constituting large-scale simulation data, and a meta data that stores metadata (detailed information of simulation data) corresponding to each data file. A data table 32 and a use and load distribution status table 33 (hereinafter referred to as “situation table 33”) of each node that is referred to when a processing target data file group is distributed and arranged in a plurality of nodes are stored. . The file DB 31 and the metadata table 32 are created on different storage areas.

  Further, the external storage device 3 stores a program for causing the computer Y to function as a simulation data and metadata management device and to function as a control device for the parallel computer group Y (nodes # 0 to #n). ing.

The CPU 1 realizes the following functions, for example, by loading the program recorded in the external storage device 3 into the MM 2 and executing the program.
(1) Provide an input (designation) environment (UI: user interface) for parallel processing designation information using the input device 7 and the display device 8 to the user of the simulation system.
(2) Create a script and a configuration file based on the parallel processing designation information.
(3) When a script is created, metadata corresponding to each of a plurality of simulation data files (processing target data file group) designated by the user as processing targets is searched and acquired.
(4) At the time of creating a script, a node (arrangement of processing target data files) for processing each processing target data file constituting the processing target data file group is determined.
(5) The transfer control of the processing result file generated by the parallel processing of the processing target data file group and the processing target data file group is performed.

  The CPU 1 corresponds to an accepting unit, a determining unit, a control program generating unit, a file generating unit, and a determining unit according to the present invention. In addition, the CPU 1 can function as a receiving unit that accepts designation of a file identifier, an extracting unit that extracts a metadata search keyword from the file identifier, and a search unit that searches for metadata corresponding to the keyword. The external storage device 3 corresponds to a storage unit according to the present invention. The external storage device 3 functions as a metadata storage unit that stores metadata searched by the search unit.

<Parallel computer group X>
The nodes # 0 to #n constituting the parallel computer group X have the same configuration. FIG. 3 is a diagram illustrating a configuration example of a node. The node includes a CPU 11, a main memory 12, a calculation processor 13, an external storage device (for example, a hard disk) 14, and a communication interface (communication I / F) 15 that are connected to each other via a bus B 1. The communication I / F 15 is connected to the computer Y and other nodes via a network.

  The node receives the processing target data file transferred from the computer Y by the communication I / F 15 and stores it in the external storage device 14. Further, the node receives a parallel processing command and a setting file from the computer Y via the communication I / F 15.

  Then, the CPU 11 starts executing the parallel processing program stored in advance in the external storage device 3 in accordance with the description of the setting file. A calculation processor 13 is used for the calculation using the processing target data. The calculation processor 13 reads out the processing target data file stored in the external storage device 14 on the MM 12, and executes predetermined processing (for example, extraction of a predetermined area in the data file, calculation of a physical quantity) using the processing target data file. This predetermined process is executed based on the metadata.

  A processing result file is generated by a predetermined process and stored in the external storage device 14. The processing result file stored in the external storage device 14 is moved (transferred) to a predetermined storage location.

  The CPU 11 functions as a parallel processing unit that executes parallel processing according to the setting file. In addition, the CPU 11 can function as a receiving unit that receives designation of a file identifier, an extracting unit that extracts a metadata search keyword from the file identifier, and a search unit that searches for metadata corresponding to the keyword. The external storage device 14 functions as a metadata storage unit that stores metadata searched by the search unit.

<Data structure of DB and table>
Next, details of the file DB 31, the metadata table 32, and the usage and load situation table (situation table) 33 shown in FIG. 2 will be described.

<< File DB31 >>
The file DB 31 classifies and stores a large amount of simulation data files (hereinafter sometimes simply referred to as “data files”) using a directory structure.

  FIG. 4 is a diagram illustrating an example of the directory structure of the file DB 31. A directory tree starting from the root directory (the directory “data” in FIG. 4) is formed in the file DB 31, and a predetermined directory name is assigned to each directory in the hierarchy. The data file is stored in a directory located at the end of the directory tree and given a predetermined data file name.

  A data file is identified using a file identifier. The file identifier is represented by a list of names (path names) of the directories located on the path (path) of the directory tree from the root directory to the terminal directory and the data file name.

  For example, the file identifier of the data file having the data file name “timeXXX.000.000.dat” in FIG. 4 is “/data/experimentA/3D/statisticsA/variableB/timeXXX.000.000”. As described above, the file identifier includes the storage location information (file path) of the data file.

  In addition, the directory name (“3D”, “statsticsA”, “variableB”, etc.) and the data file name (“timeXXX.000.000”) in the file identifier are keywords that indicate the details (properties, etc.) of the data in the data file. It is prescribed. The keyword is composed of any one or more characters, and is arranged in at least one place in the directory name and the data file name. However, no keyword is set in the extension part of the file name. The keyword functions as a search key for searching for metadata corresponding to the processing target data.

  The data file does not necessarily need to be stored in one storage area, and may be distributed and stored in a plurality of storage areas arranged inside or outside the computer Y.

<Metadata table>
The metadata table 32 stores metadata corresponding to keywords in the file identifier. FIG. 5 is a diagram illustrating a data structure example of the metadata table 32.

  In the example illustrated in FIG. 5, the metadata table 32 includes a plurality of records storing search keys (keywords) and metadata corresponding to the search keys (keywords). The keyword is extracted as a search key from the file identifier of the data file (processing target data file) designated by the user.

  Metadata is information indicating details (properties, attributes, etc.) of simulation data (processing target data) .For example, it is information indicating physical properties of processing target data, statistical processing and time-space (vertical, horizontal, high). It may be information about time (year / month / day). For example, the information on the calculation grid shown in FIGS. 16A and 16B is information about the space. As a keyword representing such calculation grid information, for example, a variable name represented by an arbitrary number of characters is applied.

  FIG. 5 shows a case where one of the directory names included in the file identifier corresponds to one metadata. Instead of this, for example, one metadata may be searched from a combination of a plurality of keywords included in one file identifier. Further, a keyword may be included in a part of the directory name or data file name (excluding the extension), and the keyword may be extracted from the file identifier by the partial match search. A configuration in which a keyword is set only in the file path portion in the file identifier may be employed.

<< Situation Table 33 >>
FIG. 6 is a diagram illustrating an example of the data structure of the situation table 33. The situation table 33 includes a plurality of small tables 34 prepared for each node. Each small table 34 has the same data structure. The small table 34 includes identification information (user ID) of a user who is permitted to use the node, a maximum size (permitted maximum capacity) of the external storage device 14 of the node that can be used by the user, and a user currently using. It is composed of a set of records whose elements (items) are the capacity (load) of the external storage device 14. Each small table 34 is given a node identifier, and information corresponding to the node identifier is stored in the small table.

<User interface (UI)>
In the computer Y shown in FIG. 2, the CPU 1 provides an input environment (UI) for parallel processing designation information to the user of the computer Y through execution of the program.

  The user can use the UI to process the processing target data file group (file identifier), a plurality of nodes that process the processing target data file group, and the processing content for the processing target data file group (elements) of the parallel processing designation information (item). The processing type and detailed parameters), the storage location of the processing result file, etc. can be specified.

  FIG. 7 is a diagram illustrating an example of a designation screen for parallel processing designation information provided as a UI. The designation screen is displayed on the screen of the display device 8 through execution of the program by the CPU 1.

  In the example shown in FIG. 7, the designation screen includes a file path display field 81, a file list display field 82, and a command input field 83. The file path display field 81 displays a directory (file path) in the file DB 31 selected by the user using the input device 7.

  The file list display field 82 displays a list (file list) of data files (data files stored in the terminal directory in the file path) corresponding to the file path displayed in the file path display field 81. . The command input field 83 is used to input a command related to processing for the processing target data file.

  The user can operate the input device 7 to display a desired file path (select a file path) in the file path display field 81. Depending on the selection result of the file path, the display content of the file list display field 82 is changed, and the file list corresponding to the file path is displayed in the display field 82.

  The user can specify the file identifier of the processing target data file by giving a desired file name from the file list displayed in the file list display field 82 by a cursor operation using the input device 7. At this time, a plurality of data files can be designated at a time through the cursor operation. Thus, the user can specify the file identifier of the processing target data file using the file path display field 81 and the file list display field 82.

  Further, the user can designate and input the node (number of nodes) used for parallel processing, the processing content for the processing target data file group, the storage location of the processing result file, and the like using the command input field 83.

  It should be noted that when specifying the number of nodes, processing parameters, and storage position, options of the specified contents are displayed on the screen, and these are specified by the user selecting a desired option by operating the cursor. it can.

<Generation of script and setting file>
When the user designates each element of the parallel processing designation information using the UI as described above and confirms the designated content, the parallel processing designation information is stored in a parallel processing designation information file described in a predetermined format. Is stored in a predetermined position of the external storage device 3.

  FIG. 8 is a diagram illustrating a description example of the parallel processing designation information file. In FIG. 8, the parallel processing specification information file includes a computer resource specification line, a processing detail (processing content) specification line, a processing target data file, and a processing result storage position specification line corresponding thereto.

  In the computer resource designation line (first line in FIG. 8), an identifier (“NODE”) and an argument (“3” in the example in FIG. 7) indicating the number of nodes used for parallel processing are described.

  In the process detail designation line (second line in FIG. 8), the identifier (“PROC”), the process type (“PROC_A”), and the process parameter (“120.0 150.0 20.0 50.0”) indicating the process details are displayed. The argument to represent is described.

  In addition, in the designated rows (third and fourth rows in FIG. 8) of the processing target data file and the storage location, the identifier (“DATA”), the file identifier of the processing target data file, and the storage location of the corresponding processing result file are stored. Identification information (illustrated by “xxxxx” or “xxxxy”) is described. The designated line is created for each data file to be processed.

  Such a description (parallel processing designation information file) is automatically created by the CPU 1 when the user designates the number of nodes, processing contents, processing target data file group, and storage location using the UI. .

  9, 10 and 11 are flowcharts showing an example of a main routine of script and setting file creation processing executed by the CPU 1 (FIG. 2). The execution of the process is started when, for example, the creation of the parallel processing designation information file is completed or a process start instruction is input from the user.

  When the process shown in FIG. 9 is started, first, the CPU 1 performs an initialization process (step S001). Next, the CPU 1 reads the parallel processing designation information file (FIG. 8) stored in the external storage device 3 into the MM 2 (step S002).

  Next, the CPU 1 executes an analysis loop process for the parallel processing designation information. In this analysis loop process, the CPU 1 extracts specified lines one by one from the parallel processing specification information file, sets the extracted lines as analysis target lines, and analyzes the analysis target lines.

  The CPU 1 determines whether or not the analysis target row extracted from the parallel processing designation information file is a designated row of computer resources (step S003).

  At this time, if the analysis target row is a computer resource designated row (S003; YES), the CPU 1 performs parallel processing on the argument (number of nodes: “3” in the example of FIG. 8) in this analysis target row. The computer resource parameter is determined and stored in a predetermined position (a predetermined work area on MM2) (step S004). Thereafter, the CPU 1 determines the next designated line as an analysis target line, and returns the process to step S003.

  If it is determined in step S003 that the analysis target row is not a computer resource designation row (S003; NO), the CPU 1 determines whether or not the analysis target row is a processing detail designation row (step S005). .

  At this time, if the analysis target line is a process detail designation line (S005; YES), the CPU 1 designates a process type and an argument (designated process parameter: in the example of FIG. “PROC_A” (procedure A) corresponds to the processing type designation, and “120.0 150.0 20.0 50.0” corresponds to the processing parameter), and this processing type and argument are determined as processing parameters related to parallel processing, and the predetermined position (work (Area) (step S006). Thereafter, the CPU 1 determines the next designated line as an analysis target line, and returns the process to step S003.

  If it is determined in step S005 that the analysis target line is not a process detail designation line (S005; NO), the CPU 1 determines that the analysis target line is a process target data file and a storage position designation line, and follows this determination. Then, the file identifier and the storage position identification information in the analysis target line are extracted and stored in a predetermined position (work area) (S007).

  The analysis loop process described above ends when the process for the last line of the parallel processing designation information file ends. Subsequently, the CPU 1 advances the processing to step S008 in FIG.

  In step S008, the CPU 1 outputs the header portion of the parallel processing job script. The header is stored in a predetermined position in the external storage device 3 in advance as a fixed sentence. The header includes a setting file transfer instruction. In step S008, a node used for parallel processing is determined based on the data file to be processed and the number of nodes used for the specified parallel processing. The usage and load status for each of the nodes # 0 to #n is managed by, for example, the OS (operating system) of the computer Y. The number of processing data files and the number of nodes in the parallel processing designation information file are delivered to the OS.

  For example, the OS extracts nodes that are allowed to be used by the user from the nodes # 0 to #n, and considers the use of the plurality of extracted nodes, the load status, and the number of files, and the number of nodes specified. Select. For example, a specified number of nodes are determined as nodes to be used for parallel processing in the order of decreasing load from the extracted nodes. The determined usage and load status of each node is set as a small table 34 in the status table 33. As a result, the processing target data file group is processed in parallel by the specified number of nodes determined by the OS.

  The status table 33 (FIG. 6) stores a small table 34 for all the nodes # 0 to #n, and the OS refers to the small table 34 so that nodes corresponding to the designated number of nodes are arranged in order of decreasing load. The mask may be set in the small table 34 corresponding to the node that has been selected and not selected (cannot be referred to).

  Subsequently, the CPU 1 executes a loop process for analyzing and processing the processing target data file. The loop processing is executed for each file identifier (processing target data file) obtained in step S007. In this loop, the CPU 1 first identifies one of the specified processing target data file groups (processing target data file group having the file identifier obtained in step S007) (referred to as an analysis target file). Subsequently, the CPU 1 activates a subroutine for the metadata analysis processing of the analysis target file (step S009), and passes the file identifier of the analysis target file to the subroutine.

  FIG. 12 is a flowchart illustrating an example of a metadata analysis / acquisition subroutine. In FIG. 12, first, the CPU 1 accepts an input for specifying a data file (step S101). That is, the CPU 1 receives the file identifier of the analysis target file.

  Next, the CPU 1 determines whether or not the file identifier has a correct format (step S102). At this time, if the file identifier does not have the correct format (S102; NO), it is determined that the process has failed (NG), and the script and setting file creation process ends. In this case, an error display process can be performed to notify the user of the error.

  On the other hand, if the file identifier is in the correct format (S102; YES), the CPU 1 starts a keyword acquisition loop process. In the loop processing, first, the CPU 1 determines whether or not a keyword representing metadata is included in the file identifier (S103).

  For example, the CPU 1 extracts the directory name next to the root directory in the file identifier, and lists the directory name and a keyword list (keyword group stored in the metadata table 32) in the metadata table 32 (FIG. 5). And search for a keyword that matches the extracted directory name.

  At this time, if the keyword cannot be searched, the CPU 1 extracts the next directory name and collates with the keyword list. In this manner, the CPU 1 repeats the directory name or data file name extraction processing and the keyword list matching processing described above until a directory name or data file name that matches one of the keywords is found.

  If the CPU 1 finds a keyword that matches the extracted directory name or data file name (S103; YES), the CPU 1 interrupts the extraction process, and retrieves and acquires metadata corresponding to the keyword from the metadata table 32 ( Step S104).

  For example, for the file identifier “/data/experimentA/3D/statisticsA/variableB/timeXXX.000.000.dat” (FIG. 4), the above processing is performed using the metadata table 32 of the stored contents shown in FIG. When the directory name “3D” is extracted from the file identifier and collated with the keyword list, the metadata “meta01” corresponding to “3D” is acquired from the metadata table 32.

  When acquiring the metadata from the metadata table 32, the CPU 1 resumes the extraction of the directory name or data file name and the matching process with the keyword list for the file identifier. Accordingly, for example, the corresponding metadata “meta1” is acquired from the metadata table 32 using the directory name “statisticsA” next to the directory name “3D” as a keyword.

  Thereafter, the keyword acquisition loop process ends at the time when the collation process for the data file name is completed (when the matching keyword is searched, the acquisition of the corresponding metadata is completed) (S104; NO), the subroutine (S009) shown in FIG. 12 ends, and the process returns to step S010 (FIG. 10) of the main routine.

  In this way, when the user specifies the file identifier of the processing target data, the computer Y automatically specifies (acquires) the metadata corresponding to the processing target data using the property information (keyword) included in the file identifier. To do.

  In step S010, the CPU 1 analyzes the metadata, and in parallel processing of the processing target data file (analysis target file) that is the target of the loop processing, the CPU 1 relates not only to the analysis target file but also to the analysis target file. It is determined whether or not data (related data file) to be used is necessary.

  For example, when the fluid flow velocity calculation is executed in parallel processing, the velocity X component, Y component, and Z component are required. Here, if the analysis target file that is the target of the determination process in step S010 is a data file that indicates the X component of the velocity, each data file that indicates the Y and Z components is required as the related data file.

  Here, the file identifier can include a character or a character string indicating component information indicating whether it is an X component, a Y component, or a Z component in a directory name or a data file name. The file identifier of the data file of the Y component and the Z component corresponding to the data file of a certain component (for example, the X component) is a description of the character or character string of the component information in the file identifier of the data file of the X component. Created by changing. For example, if the character “X” of the component information included in the file identifier is replaced with the characters “Y” or “Z” indicating the Y component or Z component, the file of the corresponding Y component or Z component data file It becomes an identifier.

  In step S010, if the CPU 1 finds that the analysis target file is, for example, an X component data file through the analysis of the metadata obtained in step S009, it determines that a related data file is necessary (S010; YES), the process proceeds to step S011. Otherwise (S010; NO), the CPU 1 advances the process to step S012.

  In step S011, the CPU 1 generates a file identifier of the related data file. For example, as described above, the file identifier of the related data file can be generated by changing a part of the file identifier of the analysis target file. The file identifier of the generated related data file is stored in the work area on the MM 2 as a set with the file identifier of the analysis target file.

  The related data file is stored in the file DB 31 so that the actual related data file is stored on the file path indicated by the file identifier of the generated related data file. Thereafter, the process proceeds to step S012.

  In step S012, a subroutine for determining the arrangement of the analysis target file (designated data file) or the analysis target file and the related data file is executed.

  FIG. 13 is a flowchart illustrating an example of an arrangement determination subroutine (S012). In FIG. 13, when the process is started, the CPU 1 first estimates the size of the data file arranged in the node and the computer resource A required for the process (step S201).

  That is, the CPU 1 acquires the size of the analysis target file (for example, obtained from metadata). Subsequently, the CPU 1 approximates the size of the processing result file created when the processing specified by the processing detailed parameter obtained in step S006 (FIG. 9) is executed according to the corresponding metadata for the analysis target file. . The CPU 1 calculates the total value of the size of the analysis target file and the size of the processing result file as the computer resource A.

  The size of the processing result file is determined from the extraction range, for example, when the processing content specified by the processing detail parameter is processing for extracting a part of the analysis target file from the specified extraction range.

  When there is a related data file for the analysis target file, it is preferable to increase the processing efficiency that the analysis target file and the related data file are processed by the same node. For this reason, when the related data file exists in step S201, the size of the related data file and the size of the processing result file for the related data file are also included in the computer resource A. The size of the related data file and the size of the processing result file corresponding thereto can be estimated from the size of the analysis target file and the size of the processing result file corresponding thereto, for example.

  Next, the CPU 1 refers to the situation table 33 (FIG. 6), can provide the user with a capacity corresponding to the computer resource A, and is predicted to have the lightest load in the current load distribution situation. A node is searched (step S202).

  That is, the CPU 1 refers to the situation table 33 and refers to the user record in each small table 34. For example, when the use of the simulation system is started, the user ID has already been input to the computer Y by the user, and the CPU 1 refers to the record corresponding to the user ID.

  Next, the CPU 1 calculates the remaining usable size of the user in each node by subtracting the load (current used size) from the maximum size in each record. Subsequently, the CPU 1 determines the node having the largest usable size (the smallest load) as the node where the analysis target file (and the related data file) should be arranged.

  Next, the CPU 1 updates the status table 33 based on the computer resource A (step S203). That is, the CPU 1 adds the value of the computer resource A to the load value (used size) of the small table 34 corresponding to the determined node.

  For example, in the example shown in FIG. 6, when it is determined that the computer resource A of the user A (for example, 10 gigabytes) is allocated to the node # 0, the load value in the corresponding small table 34 Is updated to "20Gbyte".

  When the update of the status table 33 is finished, the CPU 1 finishes the processing of the subroutine, and passes the identifier of the node determined as the file placement destination to the main routine.

  When the process proceeds to step S013 of the main routine, the CPU 1 outputs a command statement relating to data placement on the node (referred to as “data placement command statement”).

  That is, the CPU 1 reads a template of the data arrangement command statement (stored in the external storage device 3 in advance). The template is configured so that a statement is completed when a file identifier and a node identifier to be arranged are described at predetermined positions of a fixed statement. The CPU 1 describes the identifier of the analysis target file (and related data file) at a predetermined position of the template and also describes the node identifier obtained in step S012. In this way, the completed data arrangement command becomes part of the parallel processing job script.

  Next, after completion of the parallel processing, the CPU 1 outputs a command statement (referred to as “processing result transfer command statement”) that moves the processed data (processing result file) to the storage position (step S014).

  That is, the CPU 1 reads a template of the processing result movement command statement (stored in the external storage device 3 in advance). The template is configured such that when a storage position specified by the UI is described at a predetermined position of a standard command statement, the command statement is completed. The CPU 1 writes the storage position of the processing result file for the analysis target file obtained in step S007 at a predetermined position of the template. In this way, the completed processing result movement command statement becomes a part of the parallel processing job script.

  Next, the CPU 1 stores data arrangement information (step S015). That is, the CPU 1 stores the correspondence relationship between the file identifier and the node identifier as data arrangement information in a predetermined storage area.

  If there is a file identifier of the processing target data file that is not the analysis target file when step S015 ends, the processing returns to step S009, and the above-described processing of steps S009 to S015 is executed. When the processes for the file identifiers of all the data files to be processed are completed, the process proceeds to step S016.

  By the loop processing described above, the placement destination of each processing target data file included in the processing target data file group is determined so that the load in parallel processing is minimized.

  In step S016, the CPU 1 outputs a parallel processing program execution statement. That is, the CPU 1 reads a parallel processing program execution statement stored in advance in the external storage device 3 and sets it as a part of the parallel processing job script. In this manner, a parallel processing job script including a header, a data arrangement command statement, a processing result movement command statement, and a parallel processing program execution statement is automatically generated.

  Next, the CPU 1 starts processing for creating a setting file for the parallel processing program (step S017: FIG. 11). The CPU 1 starts a parallel processing program setting creation loop process. This loop process is executed for each data file to be processed.

  When the processing is started, the CPU 1 creates a setting for the processing target data file based on the data arrangement information (correspondence between the file identifier and the node identifier) (S018).

  That is, the CPU 1 extracts a portion related to one processing target data file from the data arrangement information obtained in step S015 and combines it with the processing parameter (obtained in step S006) corresponding to this file identifier. The CPU 1 describes the result of the combination in a predetermined format for the setting file.

  The CPU 1 performs such processing for each processing target data file, and when the processing of step S019 for all processing target data files ends, the main routine ends.

  FIG. 14 is a diagram illustrating a description example of a setting file for a parallel processing program. In the example illustrated in FIG. 14, the setting file includes a plurality of lines described for each processing target data file.

  In each line, a node identifier, a process designation (in this example, “PROC_A”), a file identifier of a processing target data file, and a processing parameter are described in order from the left in FIG. Such a setting file is referred to when each node executes a parallel processing program.

<Run script>
When the creation of the script and the setting file is completed, the CPU 1 starts executing the script. By executing the script, the computer Y transfers the setting file to each node of the parallel computer group X according to the setting file transfer command statement in the header.

  In addition, the computer Y transfers each processing target data file (processing target data file group) stored in the file DB 31 to the placement destination node according to the data placement information by executing the data placement command.

  Also, the computer Y executes the processing result movement command statement, and for each node, stores the processing result file (processed data) created by processing the processing target data file at each node at the designated storage location ( For example, it is instructed to be stored in the file DB 31.

  In addition, the computer Y instructs each node to start executing the parallel processing program by executing the parallel processing program execution statement.

<Parallel processing>
Each node (FIG. 3) where the processing target data file group is arranged receives the setting file and the processing target data file from the computer Y via the network. These are stored in the external storage device 14 in the node. Thereafter, when the CPU 11 of each node receives an instruction to execute the parallel processing program from the computer Y, the CPU 11 starts executing the parallel processing program.

  FIG. 14 is a flowchart showing execution processing of a parallel processing program executed by the CPU 11. When the process shown in FIG. 14 is started, the CPU 11 first executes an initialization process (step S301). When the initialization is completed, the CPU 11 reads the setting file stored in the external storage device 14 into the MM 12 (step S302).

  Next, the CPU 11 executes a processing loop of the processing target data file according to the setting file. In this processing loop, the CPU 11 sets one line in the setting file as a processing target line, and executes processing for the processing target data file according to the setting contents described in the processing target line.

  In the loop, first, the CPU 11 refers to the node identifier in the setting file, and determines whether or not this node identifier is equal to the identifier of the own node (step S303).

  At this time, when the node identifiers are not equal (S303; NO), the next line in the setting file is set as a process target line, and the process of step S303 is executed.

  On the other hand, when the node identifiers are equal (S303; YES), the CPU 11 performs a process of acquiring metadata corresponding to the file identifier described in the processing target line (step S304).

  The processing in step S304 is the same processing as the subroutine shown in FIG. That is, the CPU 11 refers to the metadata table 32A (the data structure is the same as that of the metadata table 32 (FIG. 5)) stored in the external storage device 14, and searches and acquires corresponding metadata.

  Next, the CPU 11 executes processing for the processing target data file in accordance with the processing type designation, processing parameter, and metadata in the processing target row (step S305). That is, the CPU 11 gives a processing type designation, a processing parameter, a file identifier, and metadata to the calculation processor 13. Then, the calculation processor 13 reads the processing target data file corresponding to the file identifier from the external storage device 14 to the MM 12, and executes processing according to the processing type designation and processing parameters based on the metadata.

  Thereafter, when the processing by the calculation processor 13 is completed, the CPU 11 outputs the processing result data (processed data) as a processing result file (step S306). The processing result file is transferred to, for example, the computer Y, and the computer Y stores the processing result file in a storage location designated by the user (for example, prepared in the file DB 31).

  The above-described processing is performed with each row in the setting file as a processing target row, and when the processing for all the rows is completed, the execution processing of the parallel processing program is finished.

<Modification>
In the above-described embodiment, the case where the computer Y and each node have the metadata table has been described. Instead of such a configuration, a configuration in which metadata acquired by the computer Y is transferred to each node may be applied.

  In the present embodiment, the example in which the storage area of the processing target data file (simulation data file) is provided on the external storage device 3 of the computer Y has been described. The storage area may be included in each node, and may be provided on a file server independent of the computer Y and the parallel computer group X.

<Effects of Embodiment>
According to this embodiment, when a user specifies a file identifier, the number of nodes, a processing type, a detailed processing parameter, and a storage location using an input environment (UI) for parallel processing specification information, parallel processing for a processing target data file group is performed. A control program (script) and a parallel program execution setting file are automatically created.

  In the past, when executing parallel processing, the user had to write a script of several hundred lines or more, including data file transfer control.

  According to the present embodiment, a desired script and a setting file are automatically created simply by designating or inputting information that is an element of the above-described parallel processing designation information using a UI. This can greatly reduce the user's labor. Further, since the time required for script description is reduced, the time required for obtaining the parallel processing result can be reduced. Furthermore, it is possible to eliminate the possibility of redoing parallel processing due to a description mistake of the user.

  Further, the metadata for the processing target data is automatically retrieved and acquired by designating the file identifier by the user. That is, when a user specifies a file identifier, a keyword is extracted from the file identifier, and metadata corresponding to the keyword is handled as the specified metadata. This eliminates the need for the user to input metadata designation for each data file to be processed. Therefore, it is possible to reduce the user's labor, shorten the processing time, and prevent the user's input error.

  In the automatic specification of metadata, in the present embodiment, a file identifier including data storage location information (file path) is applied to the processing target data file, and a keyword indicating the nature of the processing target data (keyword for metadata search) Is included.

  That is, data associating process target data and metadata is embedded in the file identifier. This eliminates the need to manage the associated data separately from the processing target data and the metadata. Therefore, effective use of the storage area and reduction of the management burden can be achieved. A file identifier can include a plurality of keywords.

  Further, when the user designates a processing target data file, the user designates a file identifier including a file path. Thereby, the designation of the file identifier also serves as a keyword input. Therefore, the work burden on the user can be reduced.

  Further, in the present embodiment, the metadata is configured to be stored in a storage area different from the processing target data file. As a result, the processing target data file can be efficiently stored in the storage area. Further, each node has metadata, and metadata transfer processing is eliminated. As a result, a reduction in efficiency due to the transfer of metadata can be suppressed.

  The present invention can be applied to data processing in various numerical simulation systems, for example.

Claims (10)

Accepting means for receiving parallel processing designation information including a processing target data file group, the number of a plurality of computing nodes in a parallel computer group performing parallel processing on the processing target data file group, and processing content designation for the processing target data file group When,
Storage means storing usage and load status for each of a plurality of computing nodes included in the parallel computer group;
Based on the specified number of calculation nodes and the usage and load status, the specified number of calculation nodes performing parallel processing , and each process constituting the processing target data file group for these calculation nodes A determination means for determining the arrangement of the target data file;
A data placement command for placing each processing object data file on the designated number of calculation nodes according to the placement determination result by the determining means, and the processing object for each computation node on which each processing object data file is placed Control program generation means for automatically generating a parallel processing job script including a parallel processing execution statement of the data file group;
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing A parallel processing support apparatus, comprising: a file generation unit that automatically generates a setting file including a file identifier of a target data file, an identifier of a computation node in which the processing data file is arranged, and a description of designated processing content. The determining means, for each processing target data file that constitutes the processing target data file group, from the calculation nodes of the specified number of calculation nodes included in the parallel computer group, a processing target data file and a processing result file corresponding thereto And select a computing node having a storage capacity capable of storing
2. The parallel processing support apparatus according to claim 1, wherein among the selected computation nodes, the computation node having the smallest current processing load is determined as a computation node on which the processing target data file is to be placed. The parallel processing designation information includes designation of a storage location of a processing result file generated as a result of processing on the processing target data file,
The parallel processing support apparatus according to claim 1, wherein the control program generation unit generates the control program including a command statement indicating that a processing result file is transferred to the storage location. Accepting means for receiving parallel processing designation information including a processing target data file group, the number of a plurality of computing nodes in a parallel computer group performing parallel processing on the processing target data file group, and processing content designation for the processing target data file group When,
Storage means storing usage and load status for each of a plurality of computing nodes included in the parallel computer group;
Determination means for deciding the arrangement of each processing target data file constituting the processing target data file group with respect to the specified number of calculation nodes based on the specified number of calculation nodes and the usage and load status When,
A data placement command for placing each processing object data file on the designated number of calculation nodes according to the placement determination result by the determining means, and the processing object for each computation node on which each processing object data file is placed Control program generation means for automatically generating a control program including a statement for executing parallel processing of data files, and
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing File generating means for automatically generating a configuration file including a file identifier of a target data file, an identifier of a calculation node in which the processing data file is arranged, and a description of the specified processing content ;
The data file to be processed is specified by selecting a file identifier selected from a list that displays multiple file identifiers of the data file that contains the file path of the data file stored in one of the multiple directories that make up the directory structure. Made by
Metadata storage means for storing metadata of processing target data;
A keyword list having keywords associated with the metadata;
The file path portion of the specified file identifier is used to extract a keyword for searching metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier. An extraction means for comparing a character string forming a part of the keyword list with the keyword list, and extracting a character string that matches at least one keyword in the keyword list as a keyword;
Search means for searching for metadata corresponding to the extracted keyword from the metadata storage means;
For each processing data file, in order to determine whether there is a related data file related to the processing data file, determine whether there is a related data file based on the metadata searched by the search means And a determination means,
When a processing target data file having a related data file is detected by the determination unit, the determination unit arranges the processing target data file and the related data file for the processing target data file in the same calculation node, and the control program generation unit Generating the control program for the processing target data file group including the related data file as one of the processing target data files, and the file generation means generates the setting file for the related data file.
Parallel processing support device. Receiving a parallel processing designation information including a processing target data file group, a number of a plurality of computing nodes in a parallel computer group performing parallel processing on the processing target data file group, and a processing content designation for the processing target data file group; ,
The specified number of computations for performing parallel processing based on the number of the designated computation nodes and the use and load status for each of the plurality of computation nodes included in the parallel computer group stored in the storage unit Determining the location of each processing target data file constituting the processing target data file group for the nodes and these calculation nodes ;
A data placement command for placing each processing object data file in the designated number of computation nodes according to the placement determination result in the step of determining the placement, and each computation node in which each processing object data file is placed Automatically generating and outputting a parallel processing job script including a parallel processing execution statement of the processing target data file group for :
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing A program for causing a computer to execute a step of automatically generating and outputting a setting file including a file identifier of a target data file, an identifier of a computation node in which the processing data file is arranged, and a description of a specified processing content . In the step of determining the arrangement, for each processing target data file constituting the processing target data file group, the processing target data file and the corresponding data file are calculated from the calculation nodes having the specified number of calculation nodes included in the parallel computer group. Select a computing node with a storage capacity that can store the processing result file,
6. The program according to claim 5 , wherein among the selected calculation nodes, the calculation node having the smallest current processing load is determined as a calculation node on which the processing target data file is to be arranged. The parallel processing designation information includes designation of a storage location of a processing result file generated as a result of processing on the processing target data file,
The program according to claim 5 or 6 , wherein, in the generation step of the control program, the control program including a command statement indicating that the processing result file is transferred to the storage location is generated. Receiving a parallel processing designation information including a processing target data file group, a number of a plurality of computing nodes in a parallel computer group performing parallel processing on the processing target data file group, and a processing content designation for the processing target data file group; ,
The processing for the specified number of calculation nodes based on the number of the specified calculation nodes and the use and load status for each of the plurality of calculation nodes included in the parallel computer group stored in the storage unit. Determining the location of each processing target data file constituting the target data file group;
A data placement command for placing each processing object data file in the designated number of computation nodes according to the placement determination result in the step of determining the placement, and each computation node in which each processing object data file is placed Automatically generating and outputting a control program including parallel processing execution statements of the processing target data file group for:
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing and file identifier of the target data file, the identifier of the compute nodes process data file is located, it is performed and outputting a configuration file that contains a description of the specified processing content automatically generates and, to a computer ,
The data file to be processed is specified by selecting a file identifier selected from a list that displays multiple file identifiers of the data file that contains the file path of the data file stored in one of the multiple directories that make up the directory structure. Made by
The file path portion of the specified file identifier is used to extract a keyword for searching metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier. An extraction step of comparing, as a keyword, a character string that matches at least one keyword in the keyword list, by comparing a character string that forms a part of the keyword list with a keyword list having a keyword group associated with metadata;
Searching for metadata corresponding to the extracted keyword from a metadata storage means storing metadata of processing target data;
For each processing data file, determine whether there is a related data file based on the metadata searched by the searching step to determine whether there is a related data file related to the processing data file And a determination step to perform further on the computer,
When a processing target data file having a related data file is detected in the determination step, in the determining step, the processing target data file and the related data file corresponding thereto are arranged in the same calculation node, and the control program is automatically executed. In the step of generating and outputting automatically, the control program for the processing target data file group including the related data file as one of the processing target data files is generated, and the setting file is automatically generated and output. In the step, the setting file for the related data file is generated.
program. Subject data file group, the number of the plurality of computing nodes in the parallel computer group for performing parallel processing for the processing object data file group, the step of accepting an parallelism designation information including designation of the processing content for the processing target data files And
The specified number of computations for performing parallel processing based on the number of the designated computation nodes and the use and load status for each of the plurality of computation nodes included in the parallel computer group stored in the storage unit nodes, and determining the arrangement of each processing target data files that make up the subject data file group for these compute nodes,
A data placement command for placing each processing object data file in the designated number of computation nodes according to the placement determination result in the step of determining the placement, and each computation node in which each processing object data file is placed Automatically generating and outputting a parallel processing job script including a parallel processing execution statement of the processing target data file group for :
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing Automatically generating and outputting a configuration file including a file identifier of a target data file, an identifier of a calculation node in which the processing data file is arranged, and a description of the specified processing content ;
A parallel processing support method including: Subject data file group, the number of the plurality of computing nodes in the parallel computer group for performing parallel processing for the processing object data file group, the step of accepting an parallelism designation information including designation of the processing content for the processing target data files And
The processing for the specified number of calculation nodes based on the number of the specified calculation nodes and the use and load status for each of the plurality of calculation nodes included in the parallel computer group stored in the storage unit. determining the arrangement of each processing target data files that constitute the target data file group,
A data placement command for placing each processing object data file in the designated number of computation nodes according to the placement determination result in the step of determining the placement, and each computation node in which each processing object data file is placed Automatically generating and outputting a control program including parallel processing execution statements of the processing target data file group for:
Wherein a configuration file for parallel processing by each computing node in which each subject data file is located is referred to when performing processing of the processing target data files that are placed on itself, for each processing target data file, processing Automatically generating and outputting a configuration file including a file identifier of the target data file, an identifier of a calculation node in which the processing data file is arranged, and a description of the specified processing content , and
The data file to be processed is specified by selecting a file identifier selected from a list that displays multiple file identifiers of the data file that contains the file path of the data file stored in one of the multiple directories that make up the directory structure. Made by
The file path portion of the specified file identifier is used to extract a keyword for searching metadata corresponding to the data file specified by the specified file identifier from the file path portion of the specified file identifier. An extraction step of comparing, as a keyword, a character string that matches at least one keyword in the keyword list, by comparing a character string that forms a part of the keyword list with a keyword list having a keyword group associated with metadata;
Searching for metadata corresponding to the extracted keyword from a metadata storage means storing metadata of processing target data;
Determining whether there is a related data file based on the metadata retrieved by the search to determine whether there is a related data file related to the processed data file for each processed data file And further comprising steps
When a processing target data file having a related data file is detected in the determination step, in the determining step, the processing target data file and the related data file corresponding thereto are arranged in the same calculation node, and the control program is automatically Generating and outputting the control program for the processing target data file group including the related data file as one of the processing target data files, and automatically generating and outputting the setting file. In the step, the setting file for the related data file is generated.
Parallel processing support method.

Images