JP7529739B2 - Model updating method and model updating system - Google Patents

Description

The present invention relates to a model updating method and a model updating system.

Traditionally, in order to improve the accuracy of a trained learning model constructed by learning data, it has been common to add input information (features, explanatory variables, etc.) to the learning model, retrain the learning model, evaluate its performance, and repeat the retraining and performance evaluation until an improvement in accuracy is confirmed.

Alice Zheng and Amanda Casari, translated by Hoksoem Co., Ltd., "Feature Engineering for Machine Learning: Principles and Practice with Python (O'Reilly Japan)", Ohmsha, February 23, 2019

However, in the above-mentioned conventional technology, in order to improve the accuracy of a trained learning model, input information is added and the learning model is reconstructed from scratch, which incurs the cost of reconstruction (computational load and calculation time) while not necessarily improving the accuracy of the learning model. Furthermore, in the analysis of complex time-series data spanning multiple phases, there is a risk that a learning model with sufficient accuracy cannot be constructed even if learning is performed over a certain period spanning multiple phases. In other words, there is a problem in that while the cost of improving the accuracy of a trained learning model is high, it is not necessarily possible to expect a sufficient improvement in accuracy that is commensurate with the cost.

The present invention has been made in consideration of the above-mentioned background, and aims to sufficiently improve the accuracy of a trained learning model at low cost.

In one aspect of the present invention, a model updating method for updating a model for estimating an output of a system or an observed value of an actual phenomenon, which is executed by a model updating device, includes a model error calculation step for calculating a model error based on a first error, which is the difference between the observed value and first output information, which is an estimate of the observed value outputted by inputting first input information into the model, an additional model error calculation step for calculating an additional model error based on a second error, which is the difference between the first error and second output information, which is the difference between the first error and second input information different from the first input information, which is inputted into an additional model candidate represented by a predetermined function including a predetermined parameter, and a specific value of the predetermined parameter that minimizes the additional model error. and an additional model error minimum value calculation step for calculating the minimum value of the additional model error when the specific value is the specified value; a model improvement degree calculation step for calculating a model improvement degree that is the difference between the model error and the minimum value of the additional model error; a model improvement degree determination step for determining whether the model improvement degree satisfies a specified condition; and a model update step for updating the model by adding the additional model candidate into which the specific value is substituted as an additional model to the model so that the model receives the first input information and the second input information as inputs and outputs the sum of the first output information and the second output information when it is determined that the model improvement degree satisfies the specified condition.

According to one aspect of the present application, the accuracy of a trained learning model can be sufficiently improved at low cost.

FIG. 2 is a diagram for explaining an overview of a model update according to the embodiment. FIG. 1 is a diagram showing the configuration of a model generation system according to an embodiment. 1 is a flowchart showing a model generation process according to the embodiment. FIG. 1 is a diagram showing the configuration of a model updating system according to an embodiment. FIG. 13 is a diagram showing a screen for determining a model improvement degree judgment condition according to the embodiment. 11 is a flowchart showing a model update process according to the embodiment. FIG. 1 is a diagram showing a configuration of a prediction system according to an embodiment. FIG. 4 is a diagram showing time series data of observed values of the pollution level of sewage according to the first embodiment. FIG. 4 is a diagram showing variables correlated with the degree of contamination of sewage according to the first embodiment. FIG. 4 is a diagram showing time series data of variables correlated with the degree of sewage pollution in the first embodiment. FIG. 4 is a diagram showing time series data of the estimated value of the pollution level of sewage by the existing model according to the first embodiment. FIG. 13 is a diagram showing additional variables that are estimated to be correlated with the degree of sewage pollution in the first embodiment. FIG. 13 is a diagram showing time series data of additional variables estimated to have a correlation with the degree of sewage pollution in Example 1. FIG. 13 is a diagram showing time series data of observed values of transparency of an organic resin according to Example 2. FIG. 11 is a diagram showing variables correlated with the transparency of the organic resin according to Example 2. FIG. 13 is a diagram showing time series data of variables correlated with the transparency of the organic resin according to Example 2. FIG. 11 is a diagram showing time series data of the estimated value of the transparency of an organic resin by the existing model according to the second embodiment. FIG. 13 is a diagram showing additional variables that are estimated to be correlated with the transparency of the organic resin according to the second embodiment. FIG. 13 is a diagram showing time series data of additional variables that are estimated to have a correlation with the transparency of the organic resin according to Example 2. FIG. 13 is a diagram showing time series data of observed values of a personnel allocation plan at a production site according to the third embodiment. FIG. 13 is a diagram showing variables correlated with a personnel allocation plan in a production site according to the third embodiment. FIG. 13 is a diagram showing time-series data of variables correlated with a personnel allocation plan at a production site according to a third embodiment. FIG. 13 is a diagram showing time series data of estimated values of a personnel allocation plan at a production site by an existing model according to the third embodiment. FIG. 13 is a diagram showing additional variables estimated to have a correlation with a personnel allocation plan at a production site according to the third embodiment. FIG. 13 is a diagram showing time-series data of an additional variable for which a correlation with a personnel allocation plan at a production site is estimated according to a third embodiment.

Below, an embodiment of the disclosure of this application will be described with reference to the drawings. The embodiment, including the drawings, is an example for explaining this application. In the embodiment, appropriate omissions and simplifications have been made in order to clarify the explanation. Unless otherwise specified, the components of the embodiment may be singular or plural. Furthermore, a form in which one embodiment is combined with another embodiment is also included in the embodiment of this application.

The same or similar components are given the same reference numerals, and in the following embodiments and examples, their description may be omitted or only the differences may be described. Furthermore, when there are multiple identical or similar components, they may be described with different subscripts added to the same reference numerals. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. The number of each component may be singular or plural, unless otherwise specified.

In the embodiments, the processing performed by executing a program may be described. A computer performs processing defined in a program using a processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) while using memory in a main storage device and storage in an auxiliary storage device. Therefore, the processor may be the entity that performs processing by executing a program. The processor executes a program to realize a functional unit that performs processing.

Similarly, the subject of processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The subject of processing performed by executing a program may be a computing unit, and may include a dedicated circuit that performs specific processing. Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

The program may be installed on the computer from a program source. The program source may be, for example, a program distribution server or a non-transitory storage medium readable by the computer. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource (storage) that stores the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[Embodiment]
Fig. 1 is a diagram for explaining an overview of model updating according to an embodiment. In Fig. 1, a pseudo real phenomenon 100 has a real phenomenon 101 and an existing model 102 that simulates or imitates the real phenomenon 101. The real phenomenon 101 is a system that outputs a signal to be observed. It is not limited to the real phenomenon 101, but may be a system. Furthermore, the "signal" may be data. The existing model 102 is a trained existing model that simulates or imitates the real phenomenon 101.

Suppose that there are signals related to the real phenomenon 101, including an observation value y1 obtained by observing the signal output by the real phenomenon 101, one or more known signals x1, x2, ..., xn, and one or more unknown signals u1, u2, ..., um. The observation value y1, signals x1, x2, ..., xn, u1, u2, ..., um are time-series signals at times 1, 2, ..., T in the time-series data period.

Signals x1, x2, ..., xn, u1, u2, ..., um are signals whose correlation or causal relationship with observed value y1 is estimated. Unknown signals u1, u2, ..., um are signals that were not acquired or could not be observed when the existing model 102 was generated, or signals whose correlation or causal relationship with observed value y1 has not been found.

When signals x1, x2, ..., xn, u1, u2, ..., um are the explanatory variables and observed value y1 is the objective variable, some correlation may be found between the explanatory variables and the objective variable. This correlation is often used to predict observed values and perform optimization analysis. It is possible to generate an existing model 102 by learning only the time series signals of known signals x1, x2, ..., xn that are available when generating the existing model 102, and using signals x1, x2, ..., xn as inputs and outputting an estimated value y2 of observed value y1. When signals x1, x2, ..., xn are input to the existing model 102, an estimated value y2 of observed value y1 is output.

The first error e1 is the difference obtained by subtracting the estimated value y2 of the observed value y1 from the observed value y1. The model error J1 is expressed as a function of the first error e1 using a function g. The calculation or acquisition of the first error e1 and the calculation of the model error J1 are performed by a model error calculation unit 213 described later ( FIG. 4 ). The model error J1 is expressed, for example, as in Equation (1).

Here, the function g(e1) is the sum of squares error of the first error e1 over the period T of the time series data of the input signals x1, ..., xn, but it is not limited to the sum of squares error and can be any function that can evaluate the first error e1 over the period T of the time series data.

The actual phenomenon 101 and the existing model 102 may be black boxes. Even if the observed value y1 output by the actual phenomenon 101 and the estimated value y2 output by the existing model 102 are unknown, it is sufficient to obtain the first error e1.

Signals u1, ..., uk are one or more additional signals selected from unknown signals u1, u2, ..., um. The additional model 103 (additional model candidate) is represented, for example, by a linear or nonlinear function F(u1, ..., uk|θ) that uses θ as a parameter, inputs signals u1, ..., uk, and outputs an estimated value z. The additional model 103 is represented as in Equation (2). (*) ^t represents the transpose of a matrix or vector.

Furthermore, the function F(u1, ..., uk|θ) on the right side of the formula (2) may be expressed as a linear combination of functions f(ui) as in the formula (3). However, when ui = 0, f(ui) = 0. "b" on the right side of the formula (3) is a bias term.

The second error e2 is the difference obtained by subtracting the estimated value z from the first error e1. The additional model error J2 is expressed as a function of the second error e2 using a function h. The second error e2 and the additional model error J2 are calculated by an additional model error calculation unit 214 (FIG. 4), which will be described later. The additional model error J2 is expressed, for example, as in Equation (4).

Here, the function h(e2) is the squared sum error of the second error e2 over the period T of the time series data of the input signals x1, ..., xn, u1, ..., uk, but it is not limited to the squared sum error and can be any function that can evaluate the second error e2 over the period T of the time series data.

Furthermore, the minimum value J2' of the additional model error J2 is the value of the additional model error J2 when the parameter θ minimizes the additional model error J2. The minimum value J2' of the additional model error J2 is calculated by the additional model error minimum value calculation unit 215 described later (FIG. 4). When the additional model 103 is expressed in a linear form as in equation (3), the parameter θ that minimizes the additional model error J2 is obtained from the normal equation of equation (5). Furthermore, when the parameter θ that minimizes the additional model error J2 is obtained, the minimum value of the additional model error J2 when this parameter θ is obtained is also obtained. In equation (5), uj(t) represents the value of the j-th signal uj at time t of the time series data within the period T.

The model improvement degree E is the difference obtained by subtracting the minimum value J2' of the additional model error from the model error J1. The model improvement degree E is compared with a predetermined threshold or a predetermined range, and if it is equal to or greater than the predetermined threshold or outside the predetermined range, it is deemed that adding the additional model 103 to the existing model 102 will significantly improve the model accuracy. If the model improvement degree E is equal to or greater than the predetermined threshold or outside the predetermined range, the additional model 103 is added to the existing model 102 to update the model 132. The updated model 132 outputs y as a new estimated value, which is the sum of the estimated value y2 output by the existing model 102 in response to the input of signals x1, x2, ..., xn and the estimated value z output by the additional model 103 in response to the input of signals u1, ..., uk. The estimated value y is calculated by the estimation unit 311 described later (FIG. 7).

In addition, in equation (3), if all the additional signals u1, . . . , uk except for u1 are set to 0, the estimated value z is expressed as a linear function of f(u1) as in equation (6).

Similarly, when all but u2 are set to 0, all but u3 are set to 0, ..., all but uk are set to 0, zj is a linear expression of f(uj) (j = 2, ..., k). Therefore, while selecting the additional signals one by one from u1 → u2 → ... → uk, a parameter θ = [uj, b] is obtained that takes the minimum value J2' of the additional model error J2 corresponding to each additional signal. Then, this parameter θ is added to the existing model 102 in which the additional model 103 is substituted, and the model 132 can be updated sequentially. When the accuracy of the model 132 is improved by adding the additional model 103 at a certain uj among u1 → u2 → ... → uk by a certain amount, the repeated execution of the process of generating and adding the additional model 103 for ul (l = j + 1, j + 2, ...) after this uj may be terminated.

(Configuration of model generation system 1)
2 is a diagram showing a configuration of a model generation system 1 according to an embodiment. The model generation system 1 generates an existing model 102 in advance prior to an inference process using the existing model 102. The model generation system 1 includes a processor 11, a memory 12, and a storage 13.

The storage 13 stores model training data 131 and a model 132. The model training data 131 is training data used to generate the existing model 102. The model 132 includes the existing model 102.

The processor 11 has a data preprocessing unit 111, a model generation unit 112, and a model application unit 113. These functional units are realized by the processor 11 executing a predetermined program.

The data preprocessing unit 111 processes the model learning data 131 into a format suitable for training the existing model 102. The model generation unit 112 learns the model learning data 131 processed by the data preprocessing unit 111 to generate the existing model 102. The model application unit 113 saves the existing model 102 generated by the model generation unit 112 as a model 132 in the storage 13. By storing the model 132 including the existing model 102 in the storage 13, it becomes possible to deploy it in the model update system 2 and the prediction system 3 described below. The storage 13 and the model 132 may be a shared storage accessible from any of the model generation system 1, the model update system 2, and the prediction system 3.

(Model Generation Process According to the Embodiment)
3 is a flowchart showing a model generation process S1 according to the embodiment. The model generation process S1 is executed by the model generation system 1 in response to, for example, a user instruction.

First, in step S11, the data pre-processing unit 111 acquires time series data of the input signals x1, x2, ..., xn. Next, in step S12, the data pre-processing unit 111 processes the time series data of the input signals x1, x2, ..., xn acquired in step S11. Step S12 may be performed manually.

Next, in step S13, the model generation unit 112 learns the time series data of the input signals x1, x2, ..., xn processed in step S12 to generate the existing model 102. Next, in step S14, the model application unit 113 stores the existing model 102 generated in step S13 in the storage 13.

(Configuration of model updating system 2 according to the embodiment)
4 is a diagram showing a configuration of a model updating system 2 according to an embodiment. After generating an existing model 102, the model updating system 2 generates an additional model 103 to be added to the existing model 102. The model updating system 2 includes a processor 21, a memory 22, a storage 23, and an output unit 24.

The storage 23 stores additional model training data 231, the model 132, and model improvement degree judgment conditions 232. The additional model training data 231 is training data used when generating the additional models 103-1, ..., 103-r.

The model 132 includes the existing model 102 and additional models 103-1, ..., 103-r. The existing model 102 is the existing model 102 generated by the model generation system 1 shown in FIG. 2. The additional models 103-1, ..., 103-r are one or more additional models 103 that have been added to the existing model 102 because of a model improvement degree E that improves the model accuracy to a predetermined threshold value Th1 or more, and the number of added models is stored in the model 132.

The processor 21 has a data preprocessing unit 211, an additional model candidate generation unit 212, and a model error calculation unit 213. The processor 21 also has an additional model error calculation unit 214, an additional model error minimum value calculation unit 215, a model improvement degree judgment unit 216, a model update unit 217, and a model improvement degree judgment condition determination unit 218. These functional units are realized by the processor 21 executing a specified program.

The data preprocessing unit 211 processes the additional model training data 231 into a format suitable for generating the additional model 103 (additional model 103-i (i = 1, 2, ..., n)).

The additional model candidate generation unit 212 generates a model in the form of equation (2) or equation (3) that outputs an estimated value z for additional signals u1, ..., uk, using a certain function and a parameter θ. The additional model candidate generation unit 212 generates the additional model 103 using the additional model training data 231 processed by the data preprocessing unit 211.

The model error calculation unit 213 calculates the model error J1 expressed by equation (1). The additional model error calculation unit 214 calculates the additional model error J2 expressed by equation (4). The additional model error minimum value calculation unit 215 calculates a parameter θ that minimizes the additional model error J2 using equation (5), and calculates the minimum value of the additional model error J2 for this parameter θ based on equation (4).

The model improvement degree determination unit 216 calculates the model improvement degree E by subtracting the minimum value J2' of the additional model error from the model error J1. The model improvement degree determination unit 216 then compares the model improvement degree E with a predetermined threshold or a predetermined range. If the model improvement degree E is equal to or greater than the predetermined threshold or is outside the predetermined range, the model improvement degree determination unit 216 decides to add the additional model 103 under the parameter θ, which is the calculation basis for the current model improvement degree E, to the existing model 102.

When the model improvement degree determination unit 216 determines that the additional model 103 with the parameter θ, which is the basis for calculating the current model improvement degree E, is to be added to the existing model 102, the model update unit 217 adds this additional model 103 to the existing model 102 and updates the model 132.

The model improvement judgment condition determination unit 218 calculates the model improvement judgment condition 232, which is a predetermined threshold or a predetermined range for comparing the model improvement degree E, and stores it in the storage 23. The model improvement judgment condition 232 is determined by setting the value Th1 of the model improvement degree E at which the accuracy is improved by a predetermined percentage, such as 1%, 2.5%, or 5%, relative to the accuracy of the current existing model 102, as the predetermined threshold.

Alternatively, the model improvement judgment condition 232 is determined by setting the value of the model improvement E at which the accuracy of the current existing model 102 improves by a value obtained by dividing a predetermined percentage, for example 1%, 2.5%, or 5%, by the elapsed time since the previous addition of the additional model 103, as a predetermined threshold. This makes it possible to suppress frequent updates of the existing model 102 in a short period of time.

When the model improvement degree judgment condition 232 is the value Th1 or the value Th2, if the model improvement degree E is equal to or greater than the model improvement degree judgment condition 232, it is determined that the additional model 103 is added to the existing model 102 under the parameter θ, which is the calculation basis for the model improvement degree E.

Alternatively, the model improvement degree judgment condition 232 is determined by setting the range of the model improvement degree E that can be considered to be equivalent to the accuracy of the current existing model 102 as a predetermined range. In this case, if the model improvement degree E is outside the range of the model improvement degree judgment condition 232, it is determined that the additional model 103 is added to the existing model 102 under the parameter θ that is the calculation basis of the model improvement degree E.

The output unit 24 is an output device that outputs various information such as a display.

Figure 5 is a diagram showing a decision screen 24D for the model improvement degree judgment condition 232 according to the embodiment. The decision screen 24D is displayed on the display screen of the output unit 24, and displays the simulation results of the model accuracy when the model improvement degree E is increased or decreased. Based on the accuracy 24D1 of the current existing model 102, a predetermined threshold Th1 for updating the existing model 102 is set, and an additional model 103 with a model improvement degree E exceeding this threshold is set as an additional model to be added to the current existing model 102 to update it.

The accuracy 24D1 of the current existing model 102 can be calculated by model evaluation using simulation or evaluation data. The accuracy 24D1 of the current existing model 102 is data with a range and variance in the calculation results. For example, (1 + Δ/100)M, which is the maximum accuracy M of the current existing model 102 plus Δ% of M, can be set as the predetermined threshold Th1. Then, the sliding scale 24D2 is moved to change the model improvement degree E, and the range of model accuracy at each model improvement degree E is confirmed. Only the additional model 103 with a model improvement degree E that is equal to or greater than the predetermined threshold Th1 becomes an additional model to be added to the existing model 102. The example in FIG. 5 shows that when the model improvement degree E is equal to or greater than E1, the model accuracy is improved to be greater than the predetermined threshold Th1 compared to the accuracy of the current existing model 102.

When calculating the predetermined threshold Th1, various statistical values such as the mean, median, standard deviation, etc. may be used instead of the maximum value M of the accuracy of the current existing model 102. Also, when calculating the predetermined threshold Th1, Δ%/T1 (where T1 represents the time elapsed since the previous addition of the additional model 103) may be used instead of Δ%.

(Model update process S2 according to the embodiment)
6 is a flowchart showing a model updating process S2 according to the embodiment. The model updating process S2 is executed by the model updating system 2, for example, in response to detection of data drift in the existing model 102. When the model updating process S2 is executed, it is assumed that the trained existing model 102 has been deployed.

First, in step S21, the model error calculation unit 213 accepts input of the first error e1. The model error calculation unit 213 may calculate the first error e1 from the observed value y1 of the real phenomenon 101 and the estimated value y2 of the existing model 102. Next, in step S22, the model error calculation unit 213 calculates the model error J1. Next, in step S23, the model error calculation unit 213 determines whether the model error J1 exceeds the threshold value Th0. If the model error J1 exceeds the threshold value Th0 (step S23 YES), the model error calculation unit 213 proceeds to step S24, and if not (step S23 NO), the model error calculation unit 213 returns to step S21.

Step S23 is a process for monitoring whether data drift has occurred in the existing model 102. Therefore, step S23 is omitted when the purpose is to improve the model accuracy of the model 132 including the existing model 102, regardless of whether data drift has occurred in the existing model 102.

In step S24, the data preprocessing unit 211 acquires time series data of additional signals u1, ..., uk in a certain combination from the signals u1, u2, ..., um. Next, in step S25, the data preprocessing unit 211 processes the additional signals u1, ..., uk acquired in step S24 into a format suitable for generating the additional model 103. Step S25 may be performed manually.

Next, in step S26, the additional model candidate generation unit 212 generates an additional model 103 (additional model candidate) in the form of equation (2) or equation (3) that outputs an estimated value z for the additional signals u1, ..., uk processed in step S25, using a certain function and parameter θ.

Next, in step S27, the additional model error calculation unit 214 calculates the second error e2. Next, in step S28, the additional model error calculation unit 214 calculates the additional model error J2. Next, in step S29, the model improvement degree determination unit 216 calculates the model improvement degree E. Next, in step S30, the model improvement degree determination unit 216 determines whether the model improvement degree E calculated in step S29 is greater than a predetermined threshold Th1. If the model improvement degree E is greater than the predetermined threshold Th1 (step S30 YES), the model improvement degree determination unit 216 moves the process to step S31, and if the model improvement degree E is equal to or less than the predetermined threshold Th1 (step S30 NO), the model improvement degree determination unit 216 returns the process to step S24. In step S24, where the process is returned from step S20, additional signals are selected from the signals u1, u2, ..., um in a combination different from that in the previous execution of step S24, and time series data of the selected additional signals is obtained.

In step S31, the model update unit 217 updates the existing model 102 by adding an additional model 103 whose model improvement degree E is greater than a predetermined threshold value Th1 in step S3.

(Configuration of Prediction System 3 According to the Embodiment)
FIG. 7 is a diagram showing a configuration of a prediction system 3 according to an embodiment. The prediction system 3 uses the existing model 102 and the additional models 103-1, ..., 103-r to output an estimated value of an observed signal y1 based on an input signal. For example, the additional model 103-j (j = 1, ..., r) outputs an estimated value zj of the observed value y1 for the input signal {u(j)}. Here, {u(j)} is a set consisting of a certain combination of signals selected from signals u1, u2, ..., um, and {u(j)} ∩ {u(j')} for j ≠ j'. The estimated value y output by the model 132 including the existing model 102 and the additional model 103-j is as shown in Equation (7). In Equation (7), the estimated value y2 is an estimated value output by the existing model 102 for the input signals x1, ..., xn. The estimated value zj is an estimated value output by the additional model 103-j for the input signal {u(j)}.

The prediction system 3 has a processor 31, a memory 32, a storage 33, and an output unit 34.

The storage 33 stores the model 132. The model 132 includes the existing model 102 and additional models 103-1, ..., 103-r. The model 132 is similar to the model 132 of the model update system 2 (Figure 4).

The processor 31 has an estimation unit 311. The estimation unit 311 is realized by the processor 31 executing a predetermined program.

The estimation unit 311 inputs input signals x1, ..., xn, {u(j)} (j = 1, ..., r) to the existing model 102 and additional model 103-j (j = 1, ..., n) included in the model 132. The estimation unit 311 also outputs an estimated value y, which is the output of the model 132, based on the estimated values y2, zj (j = 1, ..., r) output from the existing model 102 and additional model 103-j for the input signal x1, ..., xn, {u(j)} (j = 1, ..., r) in accordance with equation (7).

The output unit 34 displays the estimated value y output by the model 132 for the input signals x1, ..., xn, {u(j)} (j = 1, ..., r).

Note that two or more of the model generation system 1, the model updating system 2, and the prediction system 3 may be integrated into one system. For example, the model updating system 2 and the prediction system 3 may be integrated into one system.

(Effects of the embodiment)
According to this embodiment, the model accuracy of the model 132 including the existing model 102 can be sufficiently improved at low cost without re-learning the existing model 102 by sequentially trying to add variables and updating to add the additional model 103 only when the model accuracy improves. This embodiment can be applied not only to a model updating method for adding an additional model based on an additional signal to the existing model 102, but also to the creation of a new model. That is, after constructing the existing model 102 as the core based on the minimum necessary input signal, a new model with high accuracy can be created at low cost by sequentially adding additional variables and adding an additional model.

In this embodiment, a low-cost, highly accurate model can be provided for the model 132 used by the prediction system 3 for prediction in cases where the acquisition timing of learning data is dispersed over time, where data drift has occurred, where there is a request for additional input information, etc.

In this embodiment, learning and inference are first performed using only the additional input signals, without modifying the existing model. An additional model is added to the existing model only if it is confirmed that the accuracy is improved by adding variables. If the additional model is a linear form, it comes down to solving normal equations, so the computational cost of learning and inference is low and results can be output in a short time. Furthermore, since the additional model is derived by analytical calculations, it has the advantage of being highly explainable.

The above-mentioned "case where the acquisition of training data is dispersed over time" is, for example, when there are multiple providers of training data. In addition, for example, when the preprocessing of training data requires a large amount of man-hours, and all training data cannot be used at the same time, it is preprocessed over a period of time and becomes available sequentially. Or, for long-term analysis, the same type of observations may be acquired over a period of time.

An example of a "case where data drift has occurred" is when deterioration or other fluctuations have occurred in the actual phenomenon 101 over time. An example of a "case where there is a request for additional input information" is when there is a request to add variables in the operation of the prediction system 3.

Below, we will explain application examples of the above-mentioned embodiment, with Example 1 being the "case where the timing of acquisition of learning data is dispersed over time," Example 2 being the "case where data drift occurs," and Example 3 being the "case where there is a request for additional input information."

Example 1 shows a case where the above-mentioned embodiment is applied to a task of estimating the level of pollution of sewage using a model, as a "case where the timing of acquisition of learning data is dispersed over time." That is, model 132 including existing model 102 takes variables correlated with the level of pollution of sewage as input, and outputs an estimate of the level of pollution of sewage. Then, at a later date after the construction of existing model 102, other time series data is acquired to construct additional model 103 (candidate for additional model). Then, among additional models 103 that take newly added variables as input, additional models 103 whose model improvement level E is equal to or greater than a predetermined threshold value Th1 are added to existing model 102, and model 132 is updated.

In Example 1, the learning data of the existing model 102 and the additional model 103 are business records that require processing by dividing the data into at least two periods because the man-hours required for processing during model construction are greater than a certain amount.

Figure 8 is a diagram showing time series data of the observed value y1 of the degree of contamination of sewage in Example 1. Figure 8 shows time series data of the observed value y1 (%) of the degree of contamination of sewage at a certain observation point, which is an actual phenomenon 101 obtained in business.

Figure 9 is a diagram showing variables correlated with the level of sewage pollution in Example 1. Figure 10 is a diagram showing time series data of variables correlated with the level of sewage pollution in Example 1. Figure 11 is a diagram showing time series data of the estimated value y2 of the level of sewage pollution by the existing model 102 in Example 1. Figure 9 lists variables related to or correlated with the level of sewage pollution. Figure 10 shows time series data of the variables listed in Figure 9. Figure 11 is time series data of the estimated value y2 (%) of the level of sewage pollution output by the existing model 102 using the time series data of the variables shown in Figure 10 as input.

Figure 12 is a diagram showing additional variables that are estimated to have a correlation with the level of sewage pollution in Example 1. Figure 13 is a diagram showing time series data of additional variables that are estimated to have a correlation with the level of sewage pollution in Example 1. In Example 1, the additional variables shown in Figure 12 are of the same type as the variables shown in Figure 9, but have different acquisition dates and times. This results in a "case in which the acquisition dates of learning data are dispersed over time," and an existing model 102 is constructed based on time series data that has been acquired and processed earlier, and an additional model 103 is constructed based on time series data that has been acquired and processed later.

The time series data targeted in Example 1 is based on business records that require processing in at least two or more separate periods because the amount of data processing labor required to construct the existing model 102 and the additional model 103 (candidate additional model) is greater than a certain amount.

In the first embodiment, the observed value y1 output by the real phenomenon 101 is time-series data from business records that have already been collected. The existing model 102 is a model that has already been trained, and uses input signals x1, x2, ..., xn that have been acquired and processed simultaneously with the observed value y1 from the time-series data from business records that includes a variable that is correlated with the degree of sewage pollution, to calculate and output an estimated value y2 of the observed value y1.

The additional model 103 (additional model candidate) calculates the output signal z. The additional model 103 (additional model candidate) calculates and outputs the output signal z using additional input signals u1, ..., uk that are acquired and processed at a later date than the input signals x1, ..., xn from the time series data in the business record that includes additional variables that are estimated to be correlated with the level of sewage pollution.

The model error calculation unit 213 of the model updating system 2 measures the accuracy of the existing model 102, which was constructed based only on business records organized in the past, by calculating the model error J1 based on the first error e1 between the observed value y1 and the estimated value y2.

The additional model error calculation unit 214 of the model updating system 2 calculates the additional model error J2 based on the second error e2, which is the error between the first error e1 and the output signal z. In this way, the additional model error calculation unit 214 measures the error when an additional model 103 (additional model candidate) based on additional business records acquired and processed at a later date is added to the existing model 102.

The additional model error minimum value calculation unit 215 of the model updating system 2 finds a parameter θ that minimizes the additional model error J2. The additional model error minimum value calculation unit 215 of the model updating system 2 calculates the parameter θ that minimizes the additional model error J2 and the minimum value J2' of the additional model error J2 at that time. If the above z = F (u1, ..., uk | θ) is a linear expression, the minimum value J2' can be calculated using a normal equation.

The model improvement degree determination unit 216 of the model updating system 2 determines whether the prediction accuracy will be improved by adding the additional model 103 (additional model candidate) based on the additional business record by calculating the model improvement degree E, which is the difference between the model error J1 and the minimum value J2'. If the model improvement degree determination unit 216 determines that the model improvement degree E is greater than a predetermined threshold value Th1, the model updating unit 217 of the model updating system 2 adds the additional model 103 (additional model candidate) to the existing model 102, assuming that the prediction accuracy will be improved.

When the additional model 103 (additional model candidate) is added to the existing model 102, the estimation unit 311 of the prediction system 3 outputs an estimate y of the observed value y1. The estimate y is the sum of the estimate y2 output by the existing model 102 for the input signals x1, ..., xn, and the output signal z output by the additional model 103 for the additional input signals u1, ..., uk.

In the first embodiment, an existing model 102 is first constructed based on a portion of time-series data based on business records that take a long time to process, and then, if the model accuracy improves, an additional model 103 and additional variables are added based on the remaining business records or added business records. This speeds up the construction of the model and the start-up of the prediction system, while reducing the time required to process time-series data based on business records, making it possible to create a highly accurate model at low cost.

Example 2 shows a case where data drift occurs, in which the above-mentioned embodiment is applied to an estimation task of the transparency of an organic resin using a model at a manufacturing site. That is, model 132 including existing model 102 takes variables correlated with the transparency of the organic resin as input, and outputs an estimated value of the transparency of the organic resin. Then, when data drift occurs after a certain period of time has passed since the construction of existing model 102, a different type of time series data is acquired from a new sensor installed on the production line to construct additional model 103 (additional model candidate). Then, among the additional models 103 that take the newly added variables as input, the additional models 103 whose model improvement degree E is equal to or greater than a predetermined threshold value Th1 are added to existing model 102, and model 132 is updated.

In Example 2, the learning data for the existing model 102 and the additional model 103 is information regarding the quality control of products at the manufacturing site.

Figure 14 is a diagram showing time series data of observed values of transparency of organic resin in Example 2. Figure 14 shows time series data of observed values y1 (%) of transparency of organic resin at a certain manufacturing site, which is an actual phenomenon 101 obtained in business.

Figure 15 is a diagram showing variables correlated with the transparency of organic resin in Example 2. Figure 16 is a diagram showing time series data of variables correlated with the transparency of organic resin in Example 2. Figure 17 is a diagram showing time series data of the estimated value of transparency of organic resin by the existing model in Example 2. Figure 15 lists variables related to or correlated with the degree of sewage pollution. Figure 16 shows time series data of the variables listed in Figure 15. Figure 17 is time series data of the estimated value y2 (%) of transparency of organic resin output by the existing model 102 using the time series data of the variables shown in Figure 15 as input.

Figure 18 is a diagram showing additional variables estimated to have a correlation with the transparency of the organic resin according to Example 2. Figure 19 is a diagram showing time series data of additional variables estimated to have a correlation with the transparency of the organic resin according to Example 2. In Example 2, the additional variables shown in Figure 18 are of a different type compared to the variables shown in Figure 15. Therefore, in a case where "data drift has occurred" in the existing model 102 constructed based on previously acquired and processed time series data, an additional model 103 is constructed based on time series data acquired and processed later via a newly installed additional sensor.

In Example 2, the observed value y1 output by the real phenomenon 101 is time-series data on the quality of products produced at the manufacturing site. The existing model 102 is a model that has already been trained. The existing model 102 receives input signals x1, ..., xn that have been acquired and processed simultaneously with the observed value y1 from the time-series data on the quality of products produced at the manufacturing site, which includes a variable correlated with the transparency of the organic resin, and calculates and outputs an estimated value y2 of the observed value y1.

The additional model 103 (additional model candidate) calculates the output signal z. The additional model 103 (additional model candidate) acquires, processes, and uses input signals u1, ..., uk other than input signals x1, x2, ..., xn from time-series data related to the quality of products produced at the manufacturing site, including additional variables estimated to be correlated with the transparency of the organic resin. The input signals u1, ..., uk are acquired by adding sensors, etc. In other words, the additional model 103 (additional model candidate) calculates and outputs the output signal z when a deviation occurs between the output signal y1 and the estimated value y2 due to equipment deterioration after a certain period of time, changes in temperature, etc.

The model error calculation unit 213 of the model updating system 2 calculates a model error J1 based on a first error e1 between the observed value y1 and the estimated value y2. In this way, the model error calculation unit 213 measures the accuracy of the existing model 102 that was constructed based only on the time-series data regarding the quality of products at the manufacturing site that was acquired up to the start time of the data analysis.

The additional model error calculation unit 214 of the model updating system 2 calculates the additional model error J2 based on the second error e2, which is the error between the first error e1 and the output signal z. As a result, the additional model error calculation unit 214 measures the error when an additional model 103 (additional model candidate) based on additional time series data acquired and processed by adding a sensor or the like is added to the existing model 102.

The additional model error minimum value calculation unit 215 of the model updating system 2 determines the parameter θ that minimizes the additional model error J2. The additional model error minimum value calculation unit 215 of the model updating system 2 calculates the parameter θ that minimizes the additional model error J2 and the minimum value J2' of the additional model error J2 at that time.

The model improvement degree determination unit 216 of the model updating system 2 obtains the model improvement degree E, which is the difference between the model error J1 and the minimum value J2'. Then, based on the model improvement degree E, the model improvement degree determination unit 216 determines whether the prediction accuracy will be improved by adding an additional model 103 (additional model candidate) based on time series data acquired by an additional sensor or the like.

When the model improvement degree determination unit 216 determines that the model improvement degree E is greater than a predetermined threshold value Th1, the model update unit 217 of the model update system 2 adds the additional model 103 (additional model candidate) to the existing model 102, assuming that the prediction accuracy will be improved.

When the additional model 103 (additional model candidate) is added to the existing model 102, the estimation unit 311 of the prediction system 3 outputs an estimate y of the observed value y1. The estimate y is the sum of the estimate y2 output by the existing model 102 for the input signals x1, ..., xn, and the output signal z output by the additional model 103 for the additional input signals u1, ..., uk.

According to the second embodiment, even if the environmental conditions change after a certain period of time has passed, a high-precision model can be created at low cost by adding new variables only if adding variables after the conditions have changed will improve accuracy.

Example 3 shows a case where "there is a request for additional input information" in which the above-mentioned embodiment is applied to a personnel allocation planning task at a production site using a model. That is, model 132 including existing model 102 takes variables correlated with the number of personnel allocated at the production site (e.g., process A) as input, and outputs an estimated value of the number of personnel allocated. Then, at a later date after the construction of existing model 102, in response to a request for additional variables from a user or the like, another time series data is acquired to construct additional model 103 (additional model candidate). Then, among additional models 103 that take the newly added variables as input, additional models 103 whose model improvement degree E is equal to or greater than a predetermined threshold value Th1 are added to existing model 102, and model 132 is updated.

In Example 3, the learning data of the existing model 102 and the additional model 103 is information regarding optimal planning in production operations.

Figure 20 is a diagram showing time series data of observed values of personnel allocation plans at a production site in Example 3. Figure 20 shows time series data of observed values y1 (people) of the optimal number of actual personnel to be allocated at a certain production site, which is an actual business phenomenon 101.

Figure 21 is a diagram showing variables correlated with the personnel deployment plan at the production site in Example 3. Figure 22 is a diagram showing time series data of variables correlated with the personnel deployment plan at the production site in Example 3. Figure 23 is a diagram showing time series data of the estimated value of the personnel deployment plan at the production site by the existing model 102 in Example 3. Figure 21 lists variables that are related or correlated with the number of personnel deployed at the production site. Figure 22 shows time series data of the variables listed in Figure 21. Figure 23 is time series data of the estimated value y2 (people) of the number of personnel deployed at the production site output by the existing model 102 using the time series data of the variables shown in Figure 21 as input.

Fig. 24 is a diagram showing additional variables estimated to have a correlation with a personnel deployment plan at a production site according to Example 3. Fig. 25 is a diagram showing time series data of additional variables estimated to have a correlation with a personnel deployment plan at a production site according to Example 3. In Example 3, the additional variables shown in Fig. 24 are different types of variables that are added in response to a request for additional variables from a user or the like, compared to the variables shown in Fig. 21. This results in a "case where there is a request for additional input information", and an existing model 102 is constructed based on time series data acquired and processed based on the previous requirements definition, and an additional model 103 is constructed based on time series data acquired and processed based on the later additional requirements definition.

In Example 3, the observed value y1 of the actual phenomenon 101 is time-series data in an optimal plan for production operations. The existing model 102 is a model that has already been trained. The existing model 102 receives input signals x1, x2, ..., xn acquired simultaneously with the observed value y1 in the time-series data in the optimal plan that includes a variable correlated with the number of personnel deployed at the production site, and calculates and outputs an estimated value y2 of the observed value y1.

The additional model 103 (additional model candidate) calculates the output signal z. The additional model 103 (additional model candidate) calculates and outputs the output signal z using input signals u1, ..., uk that are added in a later requirements definition from the time series data in the optimal plan that includes an additional variable that is estimated to be correlated with the number of personnel deployed at the production site.

The model error calculation unit 213 of the model updating system 2 measures the accuracy of the existing model 102 constructed based only on the previous requirements definition by calculating the model error J1 based on the first error e1 between the observed value y1 and the estimated value y2.

The additional model error calculation unit 214 of the model updating system 2 calculates the additional model error J2 based on the second error e2, which is the error between the first error e1 and the output signal z. In this way, the additional model error calculation unit 214 measures the error when the additional model 103 (additional model candidate) based on a later additional requirement definition is added to the existing model 102.

The additional model error minimum value calculation unit 215 of the model updating system 2 determines the parameter θ that minimizes the additional model error J2. The additional model error minimum value calculation unit 215 of the model updating system 2 calculates the parameter θ that minimizes the additional model error J2 and the minimum value J2' of the additional model error J2 at that time.

The model improvement degree determination unit 216 of the model updating system 2 determines whether the prediction accuracy will be improved by adding the additional model 103 (additional model candidate) based on the additional requirements definition by calculating the model improvement degree E, which is the difference between the model error J1 and the minimum value J2'. If the model improvement degree determination unit 216 determines that the model improvement degree E is greater than a predetermined threshold value Th1, the model updating unit 217 of the model updating system 2 adds the additional model 103 (additional model candidate) to the existing model 102, assuming that the prediction accuracy will be improved.

When the additional model 103 (additional model candidate) is added to the existing model 102, the estimation unit 311 of the prediction system 3 outputs an estimate y of the observed value y1. The estimate y is the sum of the estimate y2 output by the existing model 102 for the input signals x1, ..., xn, and the output signal z output by the additional model 103 for the additional input signals u1, ..., uk.

According to the third embodiment, even if additional variables are defined as requirements after the construction of the existing model 102, the existing model 102 is retained and the additional model 103 is constructed using only the additional variables defined later as requirements. Then, if adding the additional model 103 to the existing model 102 improves model accuracy, adding the additional variables and the additional model 103 makes it possible to create a highly accurate model at low cost and suppress a decrease in model accuracy even when variables are added in the requirements definition.

Although one embodiment of the present invention has been described above in detail, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the gist of the invention. For example, the above embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. In addition, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations.

Furthermore, the above-mentioned configurations, functional units, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC card, an SD card, a DVD, or other recording media.

In addition, in each of the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all of the control lines and information lines in the implementation. For example, in reality, it can be considered that almost all components are interconnected.

The above-described layout of the various functional units, processing units, and databases of the model generation system 1, model updating system 2, and prediction system 3 is merely an example. The layout of the various functional units, processing units, and databases can be changed to an optimal layout in terms of the performance, processing efficiency, communication efficiency, etc. of the hardware and software provided in the model generation system 1, model updating system 2, and prediction system 3.

In addition, the structure of the database (schema, etc.) that stores the various types of data mentioned above can be flexibly changed from the perspective of efficient use of resources, improved processing efficiency, improved access efficiency, improved search efficiency, etc.

1: model generation system, 2: model updating system, 3: prediction system, 102: existing model, 103, 103-1, 103-2, ...: additional models, 132: model, 211: data pre-processing unit, 212: additional model candidate generation unit, 213: model error calculation unit, 214: additional model error calculation unit, 215: additional model error minimum value calculation unit, 216: model improvement degree judgment unit, 217: model updating unit, 218: model improvement degree judgment condition determination unit, 232: model improvement degree judgment condition.

Claims (16)

A model updating method for updating a model for estimating an output of a system or an observed value of an actual phenomenon, the method comprising:
a model error calculation step of calculating a model error based on a first error which is a difference between the observed value and first output information which is an estimate of the observed value outputted by inputting first input information to the model;
an additional model error calculation step of calculating an additional model error based on a second error which is a difference between the first error and second output information outputted by inputting second input information different from the first input information to an additional model candidate represented by a predetermined function including predetermined parameters;
an additional model error minimum value calculation step of calculating a specific value of the predetermined parameter that minimizes the additional model error and a minimum value of the additional model error at the specific value;
a model improvement degree calculation step of calculating a model improvement degree which is a difference between the model error and the minimum value of the additional model error;
a model improvement degree determining step of determining whether the model improvement degree satisfies a predetermined condition;
a model updating step of adding the additional model candidate into which the specific value has been substituted as an additional model to the model so that the model receives the first input information and the second input information as input and outputs the sum of the first output information and the second output information when it is determined that the model improvement degree satisfies the predetermined condition, and updating the model;
13. A method for updating a model, comprising: 2. A model updating method according to claim 1, comprising:
the predetermined parameters are coefficients and bias terms of a linear combination;
A model updating method, characterized in that the additional model candidate is represented by the linear combination of each function inputted with each of the second input information. 3. A model updating method according to claim 2, comprising:
The additional model candidate is expressed in the form of a linear function of a function to which one of the second input information is input,
a step of selecting the second input information one by one in the additional model error calculation step, calculating the additional model error based on the selected one piece of second input information, and repeatedly executing the additional model error minimum value calculation step, the model improvement degree calculation step, the model improvement degree determination step, and the model updating step. 4. A model updating method according to claim 3, comprising the steps of:
a step of terminating the repeated execution when a certain level of improvement in model accuracy is obtained when the additional model candidate for a certain piece of second input information is added to the model. 2. A model updating method according to claim 1, comprising:
2. A model updating method, comprising: the predetermined condition being that the model improvement exceeds a predetermined ratio to the model error. 2. A model updating method according to claim 1, comprising:
11. A method for updating a model, comprising: the predetermined condition being that the degree of model improvement exceeds a value obtained by dividing a predetermined ratio of the model error by the elapsed time since the previous update of the model. 2. A model updating method according to claim 1, comprising:
A model updating method comprising: a first error calculation step of calculating the first error. 2. A model updating method according to claim 1, comprising:
2. A model updating method, comprising: a step of: updating a model based on a time series data of the first output information and the observed value; 2. A model updating method according to claim 1, comprising:
the additional model error is a sum-of-squares error of the second error over a data period of the time series data of the observation value, the first output information, and the second output information;
In the additional model error minimum value calculation step,
a specific value and a minimum value that minimize a square sum error of the second error are calculated using a normal equation. 2. A model updating method according to claim 1, comprising:
The model is constructed based on learning data acquired and processed during a first period,
A model updating method, characterized in that the additional model candidate is constructed based on learning data acquired and processed in a second period after the first period. 11. A model updating method according to claim 10, comprising the steps of:
the learning data is business records that need to be processed at least in separate periods into the first period and the second period because the amount of processing effort required to construct the model and the additional model candidate is greater than or equal to a certain amount. 2. A model updating method according to claim 1, comprising:
The model is constructed based on learning data acquired and processed during a first period,
the additional model candidate is constructed based on learning data acquired and processed in response to a change or transition in an environmental condition of the system or the real phenomenon during a second period that is a certain period after the first period. 13. A model updating method according to claim 12, comprising the steps of:
A model updating method, characterized in that the learning data is information regarding quality control of products at a manufacturing site. 2. A model updating method according to claim 1, comprising:
The model is constructed based on learning data acquired and processed based on a requirements definition defined in a first period,
a model updating method, characterized in that the additional model candidate is constructed based on learning data acquired and processed based on an additional requirement definition defined in a second period after the first period. 15. A model updating method according to claim 14, comprising the steps of:
A model updating method, wherein the learning data is information regarding an optimal plan for a production operation. A model updating system for updating a model for estimating an output of a system or an observed value of an actual phenomenon,
a model error calculation unit that calculates a model error based on a first error that is a difference between the observed value and first output information that is an estimate of the observed value outputted by inputting first input information to the model;
an additional model error calculation unit that calculates an additional model error based on a second error, which is a difference between the first error and second output information outputted by inputting second input information different from the first input information to an additional model candidate represented by a predetermined function including predetermined parameters;
an additional model error minimum value calculation unit that calculates a specific value of the predetermined parameter that minimizes the additional model error and a minimum value of the additional model error at the specific value;
a model improvement degree determining unit that calculates a model improvement degree, which is a difference between the model error and the minimum value of the additional model error, and determines whether the model improvement degree satisfies a predetermined condition;
a model updating unit that, when it is determined that the model improvement degree satisfies the predetermined condition, updates the model by adding the additional model candidate into which the specific value has been substituted as an additional model so that the model receives the first input information and the second input information as input and outputs the sum of the first output information and the second output information;
A model updating system comprising:

Images