JP7568592B2 - Method and computer system for evaluating models - Google Patents

Description

The present invention relates to a technique for improving training data used to generate models that perform natural language processing.

In the field of natural language processing, AI (models) are being developed that analyze text data and perform semantic interpretation and classification. Models are generated through machine learning.

In machine learning, calculations are performed using large amounts of training data. Conventional machine learning has issues in that the cost of preparing training data is high, and increasing the amount of training data does not necessarily improve the model's prediction accuracy. In order to generate a model with high prediction accuracy, it is necessary to prepare high-quality training data that has a large effect of improving prediction accuracy. In response to this, the technology described in Patent Document 1 is known.

Patent document 1 discloses that "the present invention is a learning data accuracy visualization system including a display unit 6 that performs a predetermined display, a node optimal placement calculation processing unit 2b that uses a dynamic model to calculate the optimal placement of nodes for learning data that is the subject of natural language analysis, and a display control unit 2e that controls the display unit 6 to visualize the learning data based on the calculation results by the node optimal placement calculation processing unit 2b."

JP 2019-20946 A

The technology described in Patent Document 1 visualizes the correlation between the input (question) to the model and the output (answer) from the model, helping users correct their learning data.

To improve the predictive accuracy of a model that performs natural language processing, it is necessary to acquire an algorithm that processes natural language in a similar way to how humans think. Therefore, it is necessary to improve the training data by taking into consideration not only the model output but also the correctness of the algorithm. Here, improving the training data is a concept that includes both correcting the training data and adding training data.

In addition, the predictive accuracy of a model also depends on the quality of the test data used to evaluate the model, so improving the test data is also important.

Conventional techniques do not take into account the correctness of the algorithm, nor do they take into account the improvement of test data.

The present invention aims to provide a system and method that supports the improvement of training data and test data by taking into account the model output and algorithm correctness.

A representative example of the invention disclosed in the present application is as follows. That is, a method for evaluating a model executed by a computer system, which accepts a document as input and outputs a classification label by executing a task using the document, the computer system includes at least one computer having a processor, a storage device connected to the processor, and an interface connected to the processor, and holds judgment basis information that manages the association between the classification label and a keyword of interest that is considered important when a person assigns the classification label, and the method for evaluating the model includes a first step in which the at least one computer inputs test data including an input document and a correct classification label to the model, and a second step in which the at least one computer determines whether the model is correct. The method includes a second step of identifying the basis keywords in the input document that were considered important when outputting the classification label, a third step of the at least one computer judging the correctness of the output of the model based on the classification label and the correct classification label, a fourth step of the at least one computer judging the correctness of the model's judgment basis based on the basis keywords and the keyword of interest corresponding to the classification label, and a fifth step of the at least one computer presenting an evaluation result including information that serves as a guideline for improving the training data and test data used to retrain the model based on the output of the model and the judgment result of the correctness of the model's judgment basis.

According to the present invention, by presenting improvement guidelines for training data and test data that take into account the correctness of the model output and the judgment basis, it is possible to support the improvement of training data and test data. By improving at least one of the training data and test data based on the improvement guidelines, the user can generate a model with high prediction accuracy.

FIG. 1 illustrates an example of a system configuration according to a first embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer that configures a computer system according to a first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of learning data according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of test data according to the first embodiment. FIG. 1 is a diagram showing an example of a data structure of information for managing a learning dataset in the first embodiment. FIG. 2 is a diagram illustrating an example of a data structure of information for managing a test data set in the first embodiment. FIG. 13 is a diagram illustrating an example of a data structure of data stored in determination basis information according to the first embodiment. 11 is a diagram illustrating an example of a data structure of data stored in model management information according to the first embodiment. FIG. 11 is a diagram illustrating an example of a data structure of data stored in model management information according to the first embodiment. FIG. 11 is a diagram illustrating an example of a data structure of data stored in model management information according to the first embodiment. FIG. FIG. 2 is a sequence diagram illustrating a process flow of the system according to the first embodiment. 1 is a flowchart illustrating a learning/evaluation process executed by the computer system of the first embodiment.

The following describes an embodiment of the present invention with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiment shown below. Those skilled in the art will easily understand that the specific configuration can be changed without departing from the concept or spirit of the present invention.

In the configuration of the invention described below, the same or similar configurations or functions are given the same reference symbols, and duplicate explanations are omitted.

FIG. 1 is a diagram showing an example of the configuration of a system according to the first embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a computer constituting a computer system according to the first embodiment. FIG. 3A is a diagram showing an example of the data structure of training data according to the first embodiment. FIG. 3B is a diagram showing an example of the data structure of test data according to the first embodiment.

The system is composed of a computer system 100 and a terminal 101. The computer system 100 and the terminal 101 are connected to each other directly or via a network (not shown). The network may be, for example, a wide area network (WAN) or a local area network (LAN), and the connection method may be either wired or wireless.

The terminal 101 is a terminal operated by a user who uses the computer system 100. The terminal 101 has a processor, memory, and a network interface, which are not shown.

The computer system 100 analyzes text data such as inquiries and reports, and executes one of the tasks of classifying the text data or judging the inclusion of two pieces of text data based on the analysis results, and outputs a classification label. In classifying the text data, for example, the business field to which the text content relates is identified, and a classification label indicating the identified field is assigned to the text data. In judging the inclusion of text data, for example, the similarity between two texts is judged, and a classification label indicating either similarity or dissimilarity is assigned to the text data.

The computer system 100 is composed of a computer 200 as shown in FIG. 2. The computer system 100 may also include a storage system and a network switch, etc.

The computer 200 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204.

The processor 201 executes a program stored in the main memory 202. The processor 201 executes processing according to the program, thereby operating as a functional unit (module) that realizes a specific function. In the following description, when a process is described with the functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit. The main memory 202 is a memory or the like, and stores the program executed by the processor 201 and the data used by the program. The main memory 202 is also used as a work area. The secondary memory 203 is a hard disk drive (HDD) or a solid state drive (SSD), etc., and stores data permanently. The program and data stored in the main memory 202 may be stored in the secondary memory 203. In this case, the processor 201 reads the program and data from the secondary memory 203 and loads them into the main memory 202. The network interface 204 communicates with other devices via a network.

The computer 200 may also have an IO interface for connecting to input devices such as a keyboard, mouse, and touch panel, as well as output devices such as a display and printer.

The computer system 100 has the following functional units: a learning unit 110, a prediction unit 111, an API 112, and a data storage unit 113.

The API 112 provides the terminal 101 with an interface for utilizing various functions of the computer system 100. The terminal 101 registers and updates datasets, executes learning processes, and the like, via the API 112. In this embodiment, the user uses the terminal 101 to input judgment basis information 120. The judgment basis information 120 is information on keywords that serve as the judgment basis for the model in the task. Details of the judgment basis information 120 will be explained using FIG. 5.

The data storage unit 113 stores and manages one or more training data sets 130 and one or more test data sets 140.

The learning dataset 130 is composed of one or more pieces of learning data 131. The learning data 131 is data used in the learning process to generate a model, and includes a data ID 301, text data 302, and a classification label 303, as shown in FIG. 3A. Note that the learning data 131 may include fields other than those described above.

Data ID 301 is a field that stores identification information of learning data 131. Text data 302 is a field that stores text data to be input to the model. When the task is classification, one piece of text data is stored in text data 302, and when the task is content judgment, two pieces of text data are stored in text data 302. Classification label 303 is a field that stores the correct answer value of the classification label output by executing a task on text data. When classifying text data, classification label 303 stores business field, etc., and when content judgment of text data is performed, classification label 303 stores similarity or dissimilarity, etc.

The test dataset 140 is composed of one or more test data 141. The test data 141 is data used to evaluate the predictive accuracy of the model, and as shown in FIG. 3B, includes a data ID 311, text data 312, and a classification label 313. Note that the test data 141 may include fields other than those described above.

Data ID 311 is a field that stores identification information of test data 141. Text data 312 is a field that stores text data to be input to the model. When the task is classification, one piece of text data is stored in text data 312, and when the task is content judgment, two pieces of text data are stored in text data 312. Classification label 313 is a field that stores the correct answer value of the classification label output by executing a task on text data. When classifying text data, classification label 303 stores business field, etc., and when judging content of text data, classification label 303 stores similarity, dissimilarity, etc.

In addition, a portion of the training data 131 may be selected from the training data set 130 to generate the test data set 140.

In the following description, when there is no need to distinguish between the training dataset 130 and the test dataset 140, they will be referred to as datasets.

The learning unit 110 executes a learning process to generate a model for performing a task. The learning unit 110 also executes an evaluation process for the model. The learning unit 110 stores the results of the learning process and the evaluation process in the model management information 121. Details of the model management information 121 will be explained using Figures 6A, 6B, and 6C.

The prediction unit 111 executes a task on the input text data using the model information stored in the model management information 121.

Regarding each functional unit of the computer system 100, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function.

FIG. 4A is a diagram showing an example of a data structure of information for managing the training dataset 130 of Example 1. FIG. 4B is a diagram showing an example of a data structure of information for managing the test dataset 140 of Example 1.

The data storage unit 113 manages the training dataset 130 using the dataset management data 400 shown in FIG. 4A, and manages the test dataset 140 using the dataset management data 410 shown in FIG. 4B. One dataset management data 400 exists for one training dataset 130, and one dataset management data 410 exists for one test dataset 140.

The dataset management data 400 includes a learning dataset name 401, a task 402, and a list 403. The dataset management data 400 may include fields other than those described above.

The learning dataset name 401 is a field that stores the name, which is the identification information of the learning dataset 130. The task 402 is a field that stores the type of task executed by the model generated using the learning dataset 130. The list 403 is a field that stores a list of the identification information of the learning data 131 that constitutes the learning dataset 130. Note that instead of a list of the identification information, the learning data 131 itself may be stored.

The dataset management data 410 includes a test dataset name 411, a task 412, and a list 413. Note that the dataset management data 410 may include fields other than those described above.

Test dataset name 411 is a field that stores the name, which is the identification information of the test dataset 140. Task 412 is a field that stores the type of task executed by the model to be evaluated. List 413 is a field that stores a list of identification information of the test data 141 that constitutes the test dataset 140. Note that instead of a list of identification information, the test data 141 itself may be stored.

Figure 5 is a diagram showing an example of the data structure of the data stored in the judgment basis information 120 in Example 1.

The judgment basis information 120 includes one or more judgment basis data 500. One judgment basis data 500 exists for one classification label. The judgment basis data 500 includes a classification label 501 and keywords 502.

Classification label 501 is a field that stores a classification label. Keywords 502 is a field that stores keywords that the user considers important when assigning a classification label. In other words, data that indicates the basis for a person's judgment is stored.

Figures 6A, 6B, and 6C are diagrams showing an example of the data structure of data stored in model management information 121 in Example 1.

The model management information 121 includes one or more model data 600. One model data 600 exists for one model. The model data 600 includes a model name 601, a task 602, model parameters 603, learning parameters 604, a learning dataset name 605, a test dataset name 606, a status 607, a valid flag 608, an accuracy evaluation index 609, and an evaluation result 610. Note that the model data 600 may include fields other than those described above.

Model name 601 is a field that stores the name, which is the identification information of the model.

Task 602 is a field that stores the type of task that the model executes.

The model parameters 603 store parameters that define the model. At the start of the learning process, the model parameters 603 store parameters of an initial model. For example, in the case of BERT (Bidirectional Encoder Representations from Transformers), the model parameters 603 store parameters of a model generated in pre-learning.

Learning parameters 604 is a field that stores hyperparameters for controlling the learning process.

The learning dataset name 605 is a field that stores the name of the learning dataset 130 used in the learning process. Note that the learning dataset name 605 may also store the number of learning data 131 obtained from the learning dataset 130.

The test dataset name 606 is a field that stores the name of the test dataset 140 used in the evaluation process. Note that the test dataset name 606 may also store the number of test data 141 obtained from the test dataset 140.

Status 607 is a field that stores a value indicating the execution status of the learning process. For example, status 607 stores "before learning," "learning," "learning completed," etc.

The valid flag 608 stores a flag indicating whether or not the model is enabled as a model to be used by the prediction unit 111. For example, the valid flag 608 stores True, which indicates that the model is enabled, and False, which indicates that the model is disabled.

The accuracy evaluation index 609 is a field that stores the value of an index for evaluating the predictive accuracy of the model. For example, the accuracy evaluation index 609 stores the accuracy rate, recall rate, precision rate, F-score, etc.

The evaluation result 610 is a field that stores the results of the evaluation process. The evaluation result 610 stores evaluation result data 650 as shown in FIG. 6B. One evaluation result data 650 exists for one test data 141.

The evaluation result data 650 includes a data ID 651, a classification label 652, a confidence level 653, a classification label correct/incorrect 654, a model judgment basis 655, and a judgment basis correct/incorrect 656. The evaluation result data 650 may include fields other than those described above.

Data ID 651 is the same field as data ID 311.

Classification label 652 is a field that stores the classification label obtained by inputting test data 141 into the model.

Confidence 653 is a field that stores the confidence (probability) of the classification label output by the model.

The classification label true/false 654 is a field that stores a value indicating whether the value of the classification label 313 in the test data 141 matches the value of the classification label 652. For example, the classification label true/false 654 stores True, which indicates that the two values match, and False, which indicates that the two values do not match.

The model judgment basis 655 is a group of fields that stores information about keywords that the model considers important when outputting a classification label. In other words, data indicating the judgment basis of the model is stored. The model judgment basis 655 includes a token list 661 and an attention list 662. As shown in FIG. 6C, the token list 661 stores a list of character strings such as words extracted from text data. The attention list 662 stores a list of the importance of character strings stored in the token list 661. The importance of a character string is a value calculated using a known technique such as an attention mechanism and SHAP (Shapley Additive exPlanations).

The judgment basis correctness 656 is a field that stores a value indicating whether the judgment basis of the user and the judgment basis of the model match. For example, the classification label correctness 654 stores True, which indicates that the judgment basis of the user and the model match, and False, which indicates that the judgment basis of the user and the model do not match.

Figure 7 is a sequence diagram that explains the processing flow of the system in Example 1.

The system of Example 1 has three processing phases: dataset registration, model generation and evaluation, and updating of the learning dataset 130.

The user operates the terminal 101 to send a dataset registration request to the computer system 100 to set at least one of the learning dataset 130 and the test dataset 140 (step S101). The dataset registration request includes the name of the dataset, the type of task, and the data to be included in the dataset.

When the data storage unit 113 receives a dataset registration request via the API 112, it registers the dataset in the storage area (step S102) and sends a completion notification to the terminal 101 (step S103). At this time, the data storage unit 113 generates at least one of the dataset management data 400 and the dataset management data 410 corresponding to the dataset.

The user operates the terminal 101 to send a learning execution request to the computer system 100 (step S111). The learning execution request includes the name of the model, the name of the learning dataset 130, the name of the test dataset 140, the learning parameters, and the judgment basis information 120.

When the learning unit 110 receives a learning execution request via the API 112, it executes a learning/evaluation process (step S112). Details of the learning/evaluation process are described using FIG. 8.

When the learning/evaluation process is completed, the learning unit 110 sends a completion notification to the terminal 101 via the API 112 (step S113).

The user operates the terminal 101 to send a request to the computer system 100 to obtain the evaluation results (step S114).

When the learning unit 110 receives an acquisition request via the API 112, it generates evaluation information and transmits the evaluation information to the terminal 101 (step S115). The evaluation information includes, for example, the value of the accuracy evaluation index and the evaluation result data 650. Note that the evaluation information may include only the evaluation result data 650 in which the classification label true/false 654 is False.

The user refers to the evaluation information and decides how to update the dataset. For example, if there are errors in both the classification label and the judgment basis of the model, the user determines that the model cannot correctly recognize the character string specified as the judgment basis, and adds learning data 131 related to the character string. On the other hand, if the judgment basis is correct but the classification label of the model is incorrect, the user determines that there is an error in the test data 141, and corrects the test data 141.

The learning unit 110 may perform the above-mentioned judgment and include the judgment result in the evaluation information.

After determining the method for updating the dataset, the user operates the terminal 101 to send a dataset update request including the update contents of the dataset to the computer system 100 (step S121).

When the data storage unit 113 receives a dataset update request via the API 112, it updates the dataset to be updated (step S122) and sends a completion notification to the terminal 101 (step S123). At this time, the data storage unit 113 updates at least one of the dataset management data 400 and the dataset management data 410 corresponding to the dataset.

After updating the dataset, the user operates the terminal 101 to send a learning execution request to the computer system 100 (step S111).

Figure 8 is a flowchart explaining the learning/evaluation process executed by the computer system 100 of the first embodiment.

The learning unit 110 generates model data 600 and stores the judgment basis information 120 in the work area (step S201).

Specifically, the learning unit 110 sets values for each of the model name 601, task 602, learning parameters 604, learning dataset name 605, and test dataset name 606 of the model data 600 based on the information included in the learning execution request. In addition, the learning unit 110 sets the status 607 of the model data 600 to "before learning."

Next, the learning unit 110 acquires learning data 131 from the learning dataset 130 specified by the user (step S202). Note that all learning data 131 in the learning dataset 130 may be acquired, or a predetermined number of learning data 131 may be acquired.

Next, the learning unit 110 executes a learning process to generate a model using the learning data 131 and the learning parameters, etc. (step S203). The learning process is a well-known technique, so a detailed description is omitted. Note that the present invention is not limited to the type of model to be learned and the learning method.

When the learning process starts, the learning unit 110 updates the status 607 of the model data 600 to "learning" and sets the information of the model generated in pre-learning to the model parameters 603. When the learning process ends, the learning unit 110 updates the status 607 of the model data 600 to "learning completed" and sets the parameters of the generated model to the model parameters 603.

Next, the learning unit 110 acquires test data 141 from the test dataset 140 specified by the user (step S204). Note that all test data 141 in the test dataset 140 may be acquired, or a predetermined number of test data 141 may be acquired.

Next, the learning unit 110 starts the evaluation process using the acquired test data 141 (step S205). The learning unit 110 selects one piece of test data 141 from the acquired test data 141.

Next, the learning unit 110 inputs test data into the model generated by the learning process and obtains output from the model (step S206). The output obtained from the model includes a classification label, a confidence level, and model judgment basis data (a token list and an attention list).

Next, the learning unit 110 identifies keywords that the model considers important based on the model judgment basis data (step S207). For example, the keyword with the highest attention value or the keyword with an attention value greater than a threshold value is identified.

Next, the learning unit 110 judges whether the classification label and the judgment basis are correct (step S208). Specifically, the following process is executed.

(S208-1) The learning unit 110 compares the value of the classification label 313 of the test data 141 with the classification label output by the model to determine whether the classification label is correct. In other words, the correctness of the model output is evaluated.

(S208-2) The learning unit 110 refers to the judgment basis information 120 and searches for judgment basis data 500 whose classification label 501 matches the classification label 313 of the test data 141. The learning unit 110 compares the identified keywords with the keywords set in the keywords 502 of the searched judgment basis data 500 to determine whether the judgment basis is correct or not. In other words, the correctness of the model's algorithm is evaluated.

For example, if the identified keyword completely matches the keyword set in keyword 502, or if the identified keyword is included in the keywords set in keyword 502, the learning unit 110 determines that the human judgment basis and the model judgment basis match.

This concludes the explanation of the processing in step S208.

Next, the learning unit 110 generates evaluation result data 650 (step S209).

Specifically, the learning unit 110 generates evaluation result data 650 in which the identification information of the test data 141 is set in the data ID 651. In addition, the learning unit 110 stores the evaluation result data 650 in the evaluation result 610 of the model data 600.

Next, the learning unit 110 determines whether processing has been completed for all acquired test data 141 (step S210).

If processing has not been completed for all acquired test data 141, the learning unit 110 returns to step S205 and executes the same processing.

When processing has been completed for all acquired test data 141, the learning unit 110 calculates an accuracy evaluation index (step S211). At this time, the learning unit 110 sets the calculated accuracy evaluation index to the accuracy evaluation index 609 of the model data 600.

Next, the learning unit 110 determines whether the termination condition is satisfied (step S212). For example, if the accuracy evaluation index is greater than a threshold value, or if the number of times the learning process has been executed is greater than a threshold value, the learning unit 110 determines that the termination condition is satisfied.

If it is determined that the termination condition is not satisfied, the learning unit 110 returns to step S202 and executes the same process. At this time, the learning unit 110 initializes the accuracy evaluation index 609 and the evaluation result 610 of the model data 600.

If it is determined that the termination condition is met, the learning unit 110 terminates the learning/evaluation process.

According to the present invention, the computer system 100 can evaluate the correctness of the model's algorithm by comparing the human judgment basis with the model's judgment basis, and present the result to the user. The user can improve the training data and test data by taking into account the model's output and the correctness of the algorithm. This makes it possible to generate a model with high predictive accuracy.

The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment describes the configuration in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. Also, some of the configurations of the embodiment can be added to, deleted from, or replaced with other configurations.

100 Computer system 101 Terminal 110 Learning unit 111 Prediction unit 112 API
113 Data storage unit 120 Judgment basis information 121 Model management information 130 Learning data set 131 Learning data 140 Test data set 141 Test data 200 Computer 201 Processor 202 Main storage device 203 Sub-storage device 204 Network interface 400, 410 Data set management data 500 Judgment basis data 600 Model data 650 Evaluation result data

Claims (8)

A method for evaluating a model, executed by a computer system, which receives a document as an input, executes a task using the document, and outputs a classification label, comprising:
The computer system comprises:
at least one computer having a processor, a storage device coupled to the processor, and an interface coupled to the processor;
retaining judgment basis information for managing the association between the classification label and a keyword of interest that is considered important when a person assigns the classification label;
The method for evaluating the model includes:
A first step in which the at least one computer inputs test data including input documents and ground truth classification labels to the model;
A second step in which the at least one computer identifies basis keywords in the input document that the model considered important when outputting the classification label;
A third step in which the at least one computer determines whether an output of the model is correct or incorrect based on the classification label and the correct classification label;
a fourth step in which the at least one computer judges whether the judgment basis of the model is correct or not based on the basis keyword and the target keyword corresponding to the classification label;
A fifth step in which the at least one computer presents an evaluation result including information serving as an improvement guideline for the training data and the test data used for re-training the model based on the output of the model and the result of determining whether the decision basis of the model is correct or incorrect;
A method for evaluating a model, comprising: A method for evaluating a model according to claim 1, comprising the steps of:
The fifth step is a method for evaluating a model, characterized in that it includes a step in which the at least one computer presents the evaluation result including information indicating the classification label to be improved based on the judgment results of the correctness of the output of the model and the correctness of the judgment basis of the model. A method for evaluating a model according to claim 1, comprising the steps of:
The second step includes:
The at least one computer obtains an index representing the importance of a keyword included in the input document in the natural language processing of the model;
and identifying, by the at least one computer, the basis keywords from among the keywords included in the input document based on the index. A method for evaluating a model according to claim 1, comprising the steps of:
A method for evaluating a model, comprising a step in which the at least one computer provides an interface for inputting the decision basis information. 1. A computer system comprising:
at least one computer having a processor, a storage device coupled to the processor, and an interface coupled to the processor;
retaining information on a model that receives a document as an input and outputs a classification label by executing a task using the document, and judgment basis information that manages associations between the classification label and a keyword of interest that is considered important when a person assigns the classification label;
The at least one computer
inputting test data including input documents and correct classification labels into the model;
Identifying basis keywords in the input document that the model considered important when outputting the classification label;
determining whether an output of the model is correct or not based on the classification label and the correct classification label;
determining whether the judgment basis of the model is correct or not based on the basis keyword and the target keyword corresponding to the classification label;
A computer system characterized by presenting evaluation results including information that serves as a guideline for improving the training data and test data used for re-learning the model based on the output of the model and the judgment results of the correctness of the model's judgment basis. 6. The computer system of claim 5,
The at least one computer presents the evaluation result including information indicating the classification label to be improved based on the judgment results of the correctness of the output of the model and the correctness of the judgment basis of the model. 6. The computer system of claim 5,
The at least one computer
In the natural language processing of the model, an index representing the importance of a keyword included in the input document is obtained;
A computer system comprising: a computer that identifies the basis keyword from among the keywords included in the input document based on the index. 6. The computer system of claim 5,
A computer system, wherein the at least one computer provides an interface for inputting the decision basis information.

Images