RU2527212C1 - Modified intelligent controller - Google Patents

Abstract

FIELD: radio engineering, communication.

SUBSTANCE: modified intelligent controller comprises a control object, a resolver, a reinforcement calculation unit, a Kalman filter correction unit, an action unit, an action selection unit, an action screening unit, a Kalman filter, Kalman filter memory and an action entry unit.

EFFECT: improved speed and adaptation parameters of operation owing to use of a Kalman filter correction unit in the device.

1 dwg

Description

The invention relates to the class of intelligent controllers that use the principle of reinforcement learning, the Kalman filter, and can be used to create control systems for objects operating in a non-deterministic environment.

Known US patent, IPC G06F 15/18 6532454 "Stable adaptive control using critic designs", which implements reinforced learning using neural networks. The device according to this patent consists of a solver simulating a neural network, a block of criticism, and also a block for calculating forecast errors and connections between blocks.

The principle of operation of the device according to the US patent, IPC G06F 15/18 6532454 is as follows - the solver (the neural network acts as it) receives the reinforcement value, calculates the action at this iteration and passes it to the modeling neural network, which calculates the predicted value of the system operating parameter. After the action is completed, the system receives the real value of the working parameter, the critic calculates the new reinforcement value and corrects the operation of the modeling neural network.

The disadvantages of the devices according to the patent of IPC G06F 15/18 6532454 are that they do not remember the history of the system and the critic works according to the initially configured parameters.

Also known is a modified intelligent controller based on the Kalman filter using the reinforcement learning principle - IPC patent G06F 15/18 2458390. This device consists of a control object, a solver, a reinforcement calculation block, an action block, an action selection block, a Kalman filter and a Kalman filter memory . The outputs of the control object are connected to the solver, the reinforcement calculation block and the action block, the outputs of the action block are connected to the solver, the Kalman filter and the action selection block, the reinforcement calculation block is connected to the action block and the Kalman filter, the Kalman filter is connected to the Kalman filter memory and the action selection block ;

the Kalman filter memory is connected to the Kalman filter, and the outputs of the action selection block are connected to the control object, Kalman filter memory and the action block.

The principle of operation of the device according to the IPC patent G06F 15/18 2458390 is as follows - the solver calculates the observed parameter of the system, the action block selects the possible actions in this situation, the Kalman filter sequentially calculates the reinforcement forecast when performing the selected actions, the action selection block selects a specific action based on the filter calculations Kalman, and feeds it to the control object. The selected action and the resulting reinforcement are entered in the action block.

The disadvantages of the devices according to the patent of IPC G06F 15/18 2458390 are the inability to change the formulas of the Kalman filter in real time when changing environmental behavior.

The task is to develop a modified intelligent controller with high speed and adaptive performance characteristics.

The technical result of the proposed device is to increase the speed and adaptive parameters.

The technical result is achieved by the fact that in a modified intelligent controller containing a control object, a solver, a reinforcement calculation unit, an action block, a Kalman filter, a Kalman filter memory, an action selection block, the first output of the control object is connected to the input of the solver, the output of the solver is connected to the first input Kalman filter, the second output of the reinforcement calculation block is connected to the second input of the Kalman filter, the first output of the Kalman filter is connected to the second input of the action selection block, the second output of the Kalman filter is connected to the first input of the Kalman filter memory, the first output of the action selection block is connected to the input of the control object, the second output of the action selection block is connected to the second input of the Kalman filter memory, the first output of the Kalman filter memory is connected to the fourth input of the Kalman filter, and an action selection block is introduced into it , a Kalman filter adjustment block and an action recording block, while the first output of the control object is also connected to the first input of the action selection block, the second output of the control object is connected to the input of the calculation block the first output of the action selection block is connected to the first input of the action block, the second output of the action selection block is connected to the first input of the action selection block and the third input of the Kalman filter, the output of the action block is connected to the second input of the action selection block, the first output of the reinforcement calculation block is connected with the first input of the action input block, the second output of the reinforcement calculation block is also connected to the first input of the Kalman filter correction block, the output of the Kalman filter correction block is connected to the fifth Kalman filter input, t the Kalman filter output is connected to the second input of the Kalman filter adjustment block, the first output of the action selection block is also connected to the second input of the action recording block, the output of the action recording block is connected to the second input of the action block.

Increasing the adaptive parameters of the device is achieved due to the fact that the Kalman filter adjustment block is introduced into the device. Separation from the block of actions of the block of selection of actions and block recording actions increases the speed characteristics of the device.

Thus, the totality of the existing features set forth in the claims, allows to achieve the desired technical result.

Figure 1 shows a diagram of a modified intelligent controller with a Kalman filter.

The system consists of several structural components: control object 1, action selection block 2, solver 3, reinforcement calculation block 4, action record block 5, action block 6, Kalman filter correction block 7, Kalman filter 8, Kalman filter memory 9 and selection block action 10.

The following connections are also present in the system: the control object is connected to the solver via communication 11, with the action selection unit for communication 12 and with the reinforcement calculation unit for 13, from the action selection unit there is a connection 14 to the Kalman filter and 15 to the action selection unit and communication 16 to the action block. From the action block there is a connection 17 to the action selection block. From the unit for calculating reinforcements, there are links 18 to the block for entering actions, 19 to the block for adjusting the Kalman filter and 20 to the Kalman filter. From the Kalman filter there is a connection to the Kalman filter adjustment block 21 and from the Kalman filter adjustment block there is a connection 22 to the Kalman filter. The solver and the Kalman filter are connected at 23. From the Kalman filter there is a connection 14 to the action selection block. From the memory of the Kalman filter there is a connection 25 to the Kalman filter and from the Kalman filter there is a connection 26 to the memory of the Kalman filter. The action selection block is connected to the Kalman filter memory via communication 27. From the action selection block, communications 28 go to the control object and 29 to the action recording block. The action recording block is connected by a connection 30 to the action block.

The action selection block 2 is designed to select all possible actions in a given situation, taking into account the minimum accumulated reinforcement for a possible action

Solver 3 is a device that implements a mathematical formula (or several formulas) that describes those variables of the control object that can be directly calculated.

The reinforcement calculation unit 4 implements a mathematical formula that calculates the real value of the reinforcement after the action (control) signal is worked out by the control object 1.

Block entry actions 5 is intended to make adjustments to the block actions. This block updates the value of the accumulated reinforcement in the cell of the selected action at the previous iteration after the action is processed by the control object.

Action block 6 stores a table of possible actions in specific situations and accumulated reinforcement for situation-action pairs.

The Kalman filter correction block 7 is designed to change the structure of the Kalman filter formulas in the event that the environment has changed significantly and the current Kalman filter settings do not allow prediction of reinforcement with sufficient accuracy.

The Kalman filter 8 is designed to calculate an unobservable quantity. The Kalman filter is performed as standard, for example, according to US patent IPC G06F 15/20 No. 5115391.

The memory of the Kalman filter 9 is intended for temporary storage of parameters of the Kalman filter block 8. The block stores as many sets of Kalman filter parameters as possible actions are selected in the action block 6.

The action selection block 10 is designed to select an action from those possible in this situation based on the "greedy rule".

The principle of operation of the intelligent controller is as follows. The control object 1 performs an action and generates a status signal (in the general case, a vector) at the output. Next, the status signal is transmitted through communication 11 to solver 3 and through communication 12, the action selection block 2. The action selection block 2, receiving the object status signal, sends a request via communication 16 to action block 6 about possible actions in this situation, indicating the minimum accumulated reinforcement , with which you can choose an action (the minimum accumulated reinforcement is indicated by the developer). The action block 6 in response generates a list of possible actions in this situation and sends them sequentially through communication 17 to the action selection block 2, which in turn sends them to the Kalman filter 8 through 14. Solver 3, receiving a signal from the control object, calculates the observed signal 23, which goes to the Kalman filter 8.

, при этом 0<ε<<1 (Sutton R., Barto A. Reinforcement Learning: An Introduction. - Cambridge: MIT Press, 1998). Выбранное действие передается по 29 на блок занесения действий 5 и по 28 объект управления 1. Блок занесения действий запоминает текущее выбранное действие, а также получает рассчитанное реальное подкрепление после отработки объектом управления предыдущего действия по 18 и заносит данное подкрепление по связи 30 в ячейку к предыдущей паре ситуация - действие в блоке действий 6. Выбранное действие также посылается в память фильтра Калмана 9 и восстанавливает параметры фильтра Калмана 8 для выбранного действия. Блок корректировки фильтра Калмана 7 включается в работу в том случае, если фильтр Калмана 8 за заданное разработчиком количество итераций ошибается в прогнозе подкрепления больше, чем на определенную величину (задается разработчиком). При этом реальное подкрепление идет по сигналу 19 от блока расчета подкрепления 4, а прогнозное значение подкрепления идет от фильтра Калмана 8 по 21. Корректировка формул фильтра Калмана 8 идет по связи 22. Корректировка параметров фильтра Калмана 8 происходит по заложенному разработчиком алгоритму.Kalman filter 8, receiving a possible action from the block selection of actions 2 and the observed signal 23 from the solver 3, calculates the possible reinforcement for each possible action. In this case, the Kalman filter, receiving a new value of the possible action, writes to the Kalman filter memory 9 its parameters by communication 26 and restores the previous ones by 25 (at the beginning of the current control cycle). After calculating the reinforcement for each possible action, the Kalman filter 8 transfers the reinforcement value of 24 to the action selection block 10, which also receives the possible actions of 15 from the action selection block 2. The action selection block 10, based on the “greedy rule”, which can be written how: with probability (1 - s) the action is selected that corresponds to the maximum reinforcement value $$R_i^{PR}\left(t+\mathrm{one}\right)$$

with 0 <ε << 1 (Sutton R., Barto A. Reinforcement Learning: An Introduction. - Cambridge: MIT Press, 1998). The selected action is transmitted by 29 to the block of entering actions 5 and by 28 of the control object 1. The block of entering actions remembers the current selected action, and also receives the calculated real reinforcement after the control object fulfills the previous action of 18 and puts this reinforcement through communication 30 into the cell to the previous pair situation - action in the action block 6. The selected action is also sent to the memory of the Kalman filter 9 and restores the parameters of the Kalman filter 8 for the selected action. The Kalman filter correction block 7 is included in the work if the Kalman 8 filter for the number of iterations specified by the developer is mistaken in the reinforcement forecast by more than a certain amount (specified by the developer). In this case, the real reinforcement comes from signal 19 from the calculation unit of reinforcement 4, and the predicted value of the reinforcement comes from Kalman filter 8 through 21. Correction of the formulas of Kalman filter 8 is made through communication 22. Correction of the parameters of the Kalman filter 8 occurs according to the algorithm laid down by the developer.

Claims (1)

A modified intelligent controller containing a control object, a solver, a reinforcement calculation unit, an action block, a Kalman filter, a Kalman filter memory, an action selection block, the first output of the control object is connected to the solver input, the solver output is connected to the first input of the Kalman filter, second output of the calculation block reinforcements are connected to the second input of the Kalman filter, the first output of the Kalman filter is connected to the second input of the action selection block, the second output of the Kalman filter is connected to the first input of the Kalman filter memory, the first the output of the action selection block is connected to the input of the control object, the second output of the action selection block is connected to the second input of the Kalman filter memory, the first output of the Kalman filter memory is connected to the fourth input of the Kalman filter, characterized in that an action selection block, a correction block of the Kalman filter are introduced into it and an action input unit, wherein the first output of the control object is also connected to the first input of the action selection unit, the second output of the control object is connected to the input of the reinforcement calculation unit, the first output of the de selection unit action is connected with the first input of the action block, the second output of the action selection block is connected with the first input of the action selection block and the third input of the Kalman filter, the output of the action block is connected with the second input of the action selection block, the first output of the reinforcement calculation block is connected with the first input of the action recording block, the second output of the reinforcement calculation block is also connected to the first input of the Kalman filter correction block, the output of the Kalman filter correction block is connected to the fifth input of the Kalman filter, the third output of the Kalman filter is connected to about the second input of the Kalman filter adjustment block, the first output of the action selection block is also connected to the second input of the action recording block, the output of the action recording block is connected to the second input of the action block.