CN112003269B - Intelligent on-line control method of grid-connected shared energy storage system - Google Patents

Abstract

The invention discloses an intelligent online control method for a grid-connected shared energy storage system, which includes building two multi-hidden layer competitive Q network models; establishing a Markov decision process of CBESS, and mapping its charging and discharging behavior into action value-based iteration Updated reinforcement learning process; determine environmental state characteristics and immediate reward function; enter E rounds of loop iterative learning; MG executes the first plan scheduling in the round, and obtains the first result obtained by the agent-perceived environment with the pre-transaction volume CBESS of the external system. a state vector s _t ; use s _t as the input in the main competitive Q network to obtain the Q value output corresponding to all actions. The remaining power SOC _t of CBESS is updated to SOC _t+1 ; MG performs secondary planning for this period according to the tradable power actually fed back by CBESS, calculates the priority values of s _t , at , r _t , and s _{t +1} , and All hyperparameters of the main competition Q network are updated through gradient backpropagation; the priority p _i of the stored data in the sumtree is updated, and the parameters of the main competition Q network are copied to the target competition Q network.

Description

Intelligent online control method of grid-connected shared energy storage system

technical field

The invention relates to the technical field of power system automation, in particular to an intelligent online control method of a grid-connected shared energy storage system.

Background technique

Different from the centrally controlled large-scale energy storage system (ESS), the shared energy storage system (CESS) is small in scale, generally only has a capacity of several MWh, and is configured in the electric transformer of the distribution substation. Secondary side to mitigate the negative effects of renewable resources and continuous load changes. Once integrated into a grid-connected microgrid (MG), CESS can improve the flexibility and reliability of the MG through fast charging and discharging. With the deregulation of the distribution market, CESS can be operated by independent entities and participate in the market and realize arbitrage through price-responsive behavior. However, in the traditional method for CESS optimization decision-making, whether it adopts centralized optimization control or decentralized coordinated optimization method, complex system modeling, data unobservability and various uncertain factors are all given to the model-based physical modeling. brings many challenges.

In recent years, machine learning has developed rapidly, and its powerful perceptual learning capabilities and data analysis capabilities meet the needs of big data applications in smart grids. Among them, Reinforcement Learning (RL) acquires environmental knowledge through the continuous interaction between the decision-making subject and the environment, and takes actions that affect the environment to achieve preset goals. Deep Learning (DL) does not rely on any analytical equations, but uses a large amount of existing data to describe mathematical problems and approximate solutions. Applying it to RL can effectively alleviate problems such as the difficulty of solving value functions. In order to solve the problems of difficult modeling, poor scalability and practicability of physical modeling methods, and at the same time overcome the difficulties of solving traditional intelligent algorithms when the state space is too large, as well as the defects of the algorithm itself, such as poor convergence, poor robustness and slow convergence speed.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide an intelligent online control method for a grid-connected shared energy storage system, comprising the following steps:

Step 1, build two multi-hidden layer competitive Q network models, the main competitive Q network and the target competitive Q network, whose input is the feature vector s _t of the observation state, and the output corresponds to the action value of at _t in each action set A Q(s _t , at _t );

Step 2, establish the Markov decision process of CBESS, map its charging and discharging behavior into a reinforcement learning process based on iterative update of action value; determine the characteristics of the environment state and the immediate reward function;

Step 3: Enter E rounds of iterative learning, and each round starts to re-initialize the load curve of MG and the output of RDG, market price and SOC of shared energy storage;

Step 4, the MG performs the first plan scheduling in the round, and obtains the first state vector s _t obtained by the agent perception environment with the pre-transaction volume CBESS of the external system;

Step 5: Use s _t as the input in the main competitive Q network to obtain the Q value output corresponding to all actions. Use the ε-greedy method to select an optimal estimated Q value in the output of the current Q value to determine its corresponding action a _t and execute it;

Step 6: Update the remaining power SOC _t of CBESS to SOC _t+1 , judge whether SOC _t+1 exceeds the range of [0,1] to determine whether it exceeds the limit, and calculate the termination judgment index done _t of this round of iterations, Simultaneously calculate the immediate reward _rt after this action;

Step 7: MG performs secondary planning for the current period according to the tradable power actually fed back by CBESS, determines the transaction power with the external system, and gives the pre-trading power Pmg.CHE t+1, Pmg.grid t+ for the next period. 1, as the perception state information of the agent in the next period; and update the state of the system to s _t+1 ;

Step 8: Calculate the priority values of s _t , at , r _t , and s _{t +1} , and store them and done _t indicators in the leaf nodes of the sumtree in turn; if the number of stored data reaches the preset small batch When sampling capacity m, randomly sample m samples from it, calculate the current target Q value and its error, and update all hyperparameters of the main competitive Q network through gradient backpropagation;

Step 9, after the Q network is updated, recalculate and update the priority p _i of the stored data in the sumtree, copy the parameters of the main competition Q network to the target Q network, and make the current state s=s _t+1 simultaneously; if s is the termination state Or when the number of iteration rounds T is reached, this round of iteration is completed, and go back to step 3 to loop; otherwise, go to step 5 to continue the iteration.

Further, the main competition Q network is a multi-hidden layer main competition Q network architecture with a state value sublayer of a single neuron and an action advantage sublayer of K neurons, and the activation function selects the ReLu function to accelerate the convergence process; The normal initialization inter-layer weight ω and the initialization bias b are constants tending to 0; the state feature vector s _t is composed of the sequence number, the state of charge of CBESS, the market electricity price, and the pre-transaction electricity of MG and CBESS/superior distribution network. As the network input, it outputs the optimal value of discrete charge and discharge actions Q _t , and finally iteratively converges by preferentially replaying data for network training.

Further, the action set A is:

Divide the action space of CBESS into K discrete charge and discharge options P(k)be, and uniformly discretize the action space A

where A is the set of all possible actions; P _be ^(k) represents the k-th charge/discharge action in the uniform discrete action space of CBESS.

Further, the described Markov decision process for establishing CBESS maps the CBESS charging and discharging behavior into a reinforcement learning process based on iterative update of action value, specifically:

The remaining power of the BESS changes continuously during the charging and discharging process, and its variation is related to the charging and discharging power and self-discharge in this period; the recursive relationship between the energy storage charging and charging is:

SoC(t)=(1-σ _sdr )·SoC(t-1)+P _be ·(1-L _c )Δt/E _cap

The energy storage discharge process is expressed as follows

SoC(t)=(1-σ _sdr )·SoC(t-1)-P _be Δt/[E _cap ·(1-L _dc )]

In the formula: SoC(t) is the state of charge of CBESS in period t; _Pbe (t) is the charge and discharge power of _CBESS in period _t ; _σsdr is the self-discharge rate of the energy storage medium; Lc and Ldc are respectively Charge and discharge losses of CBESS; Δt is the duration of each calculation window;

The maximum allowable charge and discharge power of CBESS at time t is determined by its own charge and discharge characteristics and the remaining state of charge at time t, and the constraints are satisfied during operation:

SoC _min ≤SoC(t)≤SoC _max

where: SoC _max and SoC _min are the upper and lower limits of CBESS state-of-charge constraints, respectively;

The environmental state characteristics are:

Define the environmental state feature vector perceived by CBESS at time t as s _t as

In the formula, t is the time sequence number; pric _t ^b.pre /pric _t ^s.pre respectively represent the predicted sale and purchase price of the upper power grid at time t, P _t ^mg.CHE /P _t ^mg.grid respectively represent the microgrid and CBESS Pre-traded electricity with the upper power grid;

2) The instant reward function is: CBESS obtains energy arbitrage profits by charging in off-peak hours and then discharging in peak hours; after determining the actual transaction power with the microgrid and the upper-level power grid respectively, calculate the reward income r according to the real-time price _EAP ;

The total cost of operation and maintenance of CBESS, C _o,m, is shown in the following formula

C ₁ =|P _be |·c _be

A negative return line with a coefficient σ is added as a penalty to suppress the power (P _{exc_grid} ) fluctuation of the grid connection point

r _line = -σ · |P _{exc_grid} |

If the performed action causes the SOC to exceed [0,1], a large penalty r _exc is given to prevent the agent from making unreasonable decisions in subsequent learning; the immediate reward r _t is:

Further, the MG performs the first planning and scheduling in the round, and obtains the first state vector s _t obtained by the agent perception environment of the pre-transaction volume CBESS with the external system, including the following process: For the MG model, the goal is to predict To minimize the operating cost under the price signal, the objective function of the economic dispatch model is as follows:

In the formula, T is the planning period; cCDG z is the power generation cost of the zth CDG, ci ^es is the operating cost of the _ith microgrid energy storage; PCDG z,t is the power output of the zth CDG, Pes i, t is the charging and discharging power of the i-th microgrid energy storage; Pb.gridt/Ps.grid t respectively represent the sales and purchase price of the upper distribution network in each period, and P _t ^b.CHE /P _t ^s.CHE respectively represent Electricity sales and purchase prices published by CBESS operators;

According to the forecast data, the microgrid adopts the mixed integer linear programming (MILP) method to obtain the transaction volume P _t ^mg.CHE /P _t ^mg.grid between the CBESS and the upper-level distribution network during this period, and publish the transaction information to the outside world; The agent of CBESS obtains the state feature vector s _t = [t, SOC _t , ^pric _t b.pre , pric _t ^s.pre , P _t ^mg.CHE , P _t ^mg.grid ] by sensing the external environment.

Further, in the main competitive Q network, s _t is used as the input, and the Q value output corresponding to all actions is obtained. The ε-greedy method is used to select an optimal estimated Q value in the output of the current Q value to determine its corresponding action a _t and execute it, including the following process:

In the main competitive Q network, use s _t as the input to obtain the Q value output corresponding to all actions; use the ε greedy method to select a corresponding action a _t in the current Q value output, and execute the current action a _t in the state _st ; for In the ε-greedy strategy, firstly by setting the value of ε∈(0,1), in the corresponding action, the optimal action a ^* currently regarded as the maximum Q value is greedily selected with probability (1-ε), while Randomly explore potential actions from among all K discrete optional actions with probability ε:

where ε will gradually decrease from ε _ini ε _fin over the iterative process.

Further, the remaining power SOC _t of the CBESS is updated to SOC _t+1 , and it is determined whether SOC _t+1 exceeds the range of [0,1] to determine whether it exceeds the limit, and the termination judgment index of this round of iteration is calculated based on this. done _t , and at the same time calculate the immediate reward _rt after this action, which specifically includes the following process: the power SOC _t of the CBESS is updated to SOC _t+1 , so as to judge whether this iteration is a termination state, and calculate the value after this action. Immediate reward r _t ; the binary variable done is used as the indicator for judging the termination of iteration, which is used as the interrupt indicator for each iteration process

In the formula, if the state of charge exceeds the limit during the energy storage operation, the done of this iteration is equal to 1, otherwise it is 0; done=1 means termination and jumps out of this iteration, done=0 means the iteration is not terminated.

Further, the priority values of s _t , at , r _t , and s _{t +1} are calculated in the step 8, and all of them and done _t indicators are stored in the leaf nodes of the sumtree in sequence; When the amount of data reaches the preset mini-batch sampling capacity m, randomly sample m samples from it, calculate the current target Q value and its error, and update all hyperparameters of the main competitive Q network through gradient backpropagation, where the The current target Q value y _j of is:

The proportional prioritization strategy is adopted, that is, the P(i) of the extraction probability of the i-th sample is:

where α∈[0,1] is the power exponent that converts the importance of TD error into priority; if α=0, it is converted to uniform random sampling; p _i is the priority of conversion i, calculated as follows :

p(i)=|δ _i |+ζ

为正偏差；in

is a positive deviation;

Important sampling weights are used to correct the bias, resulting in a mean square error loss function Li ( _θi ) that considers the sample priority. Finally, all parameters θ of the main competitive Q network are updated through the gradient back-propagation of the neural network:

ω _j =(N·P(j)) ^-β /max _i ω _i

θ _i =θ _{i -1} +α▽ _{θi Li} (θ _i )

where ω _j is the IS weight of sample j; β is a hyperparameter that gradually increases to 1.

The beneficial effects of the present invention are as follows: 1. The present invention endows CBESS with powerful online learning and decision-making capabilities in a high-uncertainty environment, and solves the problem of continuous and continuous environmental state by approximating the optimal action value function without relying on any analytical equation. Problems that cannot be solved iteratively due to huge space;

2. The collaborative optimization of the dual-competitive Q network structure and the priority playback strategy can effectively alleviate the problem of model overestimation, significantly improve the accuracy of surrogate decision-making and the robustness of convergence, and at the same time speed up the convergence speed of the algorithm and improve online computing. efficiency.

Description of drawings

Figure 1 is a flow chart of an intelligent online control method for a grid-connected shared energy storage system;

Fig. 2 is a schematic diagram of a competitive Q network structure;

Figure 3 is a schematic diagram of the sumtree data structure;

FIG. 4 is a schematic diagram of the algorithm structure of the priority experience playback strategy.

Detailed ways

The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

As shown in Figure 1, the invented data-driven technology for online control decision-making of a grid-connected shared energy storage system includes the following steps:

S1: Build two multi-hidden layer competitive Q network models, namely the main competitive Q network and the target competitive Q network, whose input is the feature vector s _t of the observation state, and the output corresponds to the action value of at _t in each action set A Q(s _t , at _t ). First, initialize all the parameters of the Q network, the capacity D of the data storage structure sumtree, and the priority value of its leaf nodes.

S2: Establish the Markov decision process of CBESS, map its charging and discharging behavior into a reinforcement learning process based on iterative update of action value, and determine 1) the control objective of the algorithm is: to maximize the arbitrage of the energy storage market as much as possible Suppress the power fluctuation of the grid connection point of the microgrid; 2) Combination of environmental state characteristics: including the sequence number of the current period, the remaining power of the CBESS, the predicted electricity sale/purchase price of the upper-level power grid, and the forecast of the distribution network/CBESS obtained by MG's one economic dispatch. 3) Reward function: including the energy arbitrage profit r _EAP realized by CBESS through flexible charging and discharging, the total cost of operation and maintenance C _o,m , the power fluctuation penalty r _line at the grid connection point, and the energy storage SOC violation penalty r _exc .

S3: Before the start of each round of iteration, the uncertainty data needs to be re-initialized, including the load curve of the microgrid, the output of renewable distributed generation, and the market price signal;

S4: The microgrid performs pre-planning for each period based on the forecast data, and obtains the pre-transaction amount of electricity between period t and CBESS/superior distribution network, namely P _t ^mg.CHE /P _t ^mg.grid , and publishes this information to the outside world. At the same time, the agent of CBESS obtains the state feature vector s _t = [t, SOC _t , ^pric _t b.pre , pric _t ^s.pre , P _t ^mg.CHE , P _t ^mg.grid by sensing the external environment ].

S5: Use s _t as the input in the main competitive Q network to obtain the Q value output corresponding to all actions. The ε-greedy method is used to select an optimal estimated Q value in the output of the current Q value to determine its corresponding action a _t and execute it.

S6: The remaining power SOC _t of CBESS is updated to SOC _t+1 , and it is judged whether SOC _t+1 exceeds the range of [0,1] to determine whether it exceeds the limit, and the termination judgment index done _t of this round of iteration is calculated based on this, and at the same time Calculate the immediate reward _rt after this action.

S7: MG performs secondary planning for the current period according to the tradable power actually fed back by CBESS, determines the transaction power with the external system, and gives the pre-trading power Pmg.CHE t+1, Pmg.grid t+1 for the next period , as the perception state information of the agent in the next period; at this time, the state of the system is updated to s _t+1 .

S8: Calculate the priority values of s _t , at , r _t , and s _{t +1} , and store them and done _t indicators in the leaf nodes of the sumtree in sequence. Once the number of stored data reaches the preset mini-batch sampling capacity m, it starts to randomly sample m samples from it, calculates the current target Q value and its error, and updates all the main competitive Q network through gradient backpropagation. hyperparameters.

S9: After the Q network is updated, the priority pi of the data stored in the _sumtree needs to be recalculated and updated, and the parameters of the main competing Q network are copied to the target Q network periodically, and the current state s=s _t+1 is set at the same time. If s is in the terminal state or reaches the number of iteration rounds T, the iteration of the current round is completed, and the loop is returned to S3; otherwise, it goes to step S5 to continue the iteration.

The specific process of 5.1S1 steps is:

CBESS obtains feedback and rewards by continuously sensing the power demand and market environment of the microgrid, and interacting with the environment under the control objective. A multi-hidden-layer main competition Q-network architecture is constructed with a state-value sub-layer of a single neuron and an action-dominant sub-layer of K neurons, as shown in Figure 2. The corresponding target competition Q network architecture is consistent with it. The activation function selects the ReLu function to speed up the convergence process. The normal initialization inter-layer weight ω, and the initialization bias b are both constants tending to 0. The state eigenvector s _t composed of the sequence number, the state of charge of CBESS, the market electricity price, the pre-transaction electricity of MG and CBESS/superior distribution network is used as the network input, and the optimal discrete charge and discharge action value Q _t is output, and finally passes Iteratively converges by first replaying data for network training. In this model-free reinforcement learning and data-driven intelligent decision-making method for energy storage, the priority proportional sample playback method based on the sumtree data structure is adopted. At the same time, it is compatible with DDQN, which can significantly improve the strategy accuracy and convergence speed, and increase the algorithm Robustness; at the same time, the application of competitive network architecture can prompt the agent to quickly identify the correct action during policy evaluation, with higher computational efficiency and considerable fitting accuracy, and strong adaptive ability.

Sumtree is the binary tree structure shown in Figure 3. The root node is at the topmost layer, the branch nodes are at the middle layer, and only the bottom leaf nodes are responsible for storing samples. Each parent node contains the sum of its two child nodes. Therefore, the root node is the sum of all priorities, denoted as p _total . Since this data structure provides an efficient way to compute cumulative sums of priorities, sumtrees help efficiently store, update, and sample scale variables. In the storage process, the obtained data is stored in the leaf nodes from left to right, and once the leaf nodes are filled, the old data will overflow one by one from the left. A significant advantage of this approach is that there is no need to prioritize transformations, which greatly reduces the computational burden and facilitates real-time training. Before iteration, it is necessary to determine the capacity of sumtree leaf nodes and initialize the priority value of leaf nodes.

When sensing the change of the environmental state, the agent will control the CBESS to feed back the corresponding action at _t . Divide the action space of CBESS into K discrete charge and discharge options P(k)be, and uniformly discretize the action space A

where A is the set of all possible actions; P _be ^(k) represents the k-th charge/discharge action in the uniform discrete action space of CBESS.

5.2 The specific process of the S2 step is:

The Markov decision process of CBESS is established, and the charging and discharging behavior of CBESS is mapped to the reinforcement learning process based on iterative update of action value, specifically:

The remaining power of the BESS changes continuously during the charging and discharging process, and its variation is related to the charging and discharging power and self-discharge in this period. The recursive relationship between energy storage and charging is:

SoC(t)=(1-σ _sdr )·SoC(t-1)+P _be ·(1-L _c )Δt/E _cap

The energy storage discharge process is expressed as follows

SoC(t)=(1-σ _sdr )·SoC(t-1)-P _be Δt/[E _cap ·(1-L _dc )]

Where: SoC(t) is the state of charge (SoC) of CBESS in period t; _Pbe (t) is the charge and discharge power of CBESS in period t; _σsdr is the self-discharge rate of energy storage medium; L _c and L _dc are the charging and discharging losses of CBESS, respectively; Δt is the duration of each calculation window.

The maximum allowable charge and discharge power of CBESS at time t is determined by its own charge and discharge characteristics and the remaining state of charge at time t, and the constraints are satisfied during operation:

SoC _min ≤SoC(t)≤SoC _max

Where: SoC _max and SoC _min are the upper and lower limits of the CBESS state-of-charge constraint, respectively.

Reinforcement learning is a learning that maps from the state of the environment to actions, and the goal is to maximize the cumulative reward of the agent during its interaction with the environment. RL uses Markov Decision Process (MDP) to simplify its modeling, and MDP is usually defined as a quadruple (S, A, r, f), where: S is the set of all environmental states, s _t ∈S represents the state of the agent at time t; A is the set of actions the agent can perform, at _t ∈A represents the action taken by the agent at time t; r is the reward function, r _t ~r(s _t , at _t ) represents the immediate reward value obtained by the agent performing the action a _t in the state s _t ; f is the state transition probability distribution function, s _t+1 ~ f(s _t , at _t ) indicates that the agent performs the action a _t transition in the state s _t The probability of going to the next state s _t+1 . The goal of the Markov model is to find an optimal planning policy V ^π* that maximizes the sum of expected rewards after initializing state s

In the formula, E _π represents the expectation of value under policy π; 0 < γ < 1 is a decay coefficient representing the importance of future rewards in reinforcement learning.

When the size of the problem is relatively small, the algorithm is relatively easy to solve. However, for practical problems, the state space is usually very large, the calculation cost of traditional iterative solution is too high, and there are disadvantages such as difficulty in convergence, slow convergence, and prone to over-optimal estimation. Therefore, it is necessary to use the method proposed in the present invention to improve. Solve. Corresponding to the data-driven technology of the on-line control of the grid-connected shared energy storage system proposed in the present invention, the mapping relationship is as follows:

(1) Environmental state characteristics

Define the environmental state feature vector perceived by CBESS at time t as s _t as

s _t =[t,SOC _t ^be , ^pric _t b.pre ,pric _t ^s.pre ,P _t ^mg.CHE ,P _t ^mg.grid ] ^T ,s _t ∈S

In the formula, t is the time sequence number; pric _t ^b.pre /pric _t ^s.pre respectively represent the predicted sale and purchase price of the upper power grid at time t, P _t ^mg.CHE /P _t ^mg.grid respectively represent the microgrid and CBESS Pre-traded electricity with the upper-level grid.

(2) Feedback reward

In the continuous perception and learning process of CBESS, after a given environment state s _t and a choice action a _t , the single-step immediate reward r _t obtained includes:

3) CBESS obtains energy arbitrage profit (EAP) by charging during off-peak hours and then discharging during peak hours. After determining the actual transaction power with the microgrid and the upper-level grid, respectively, the reward income r _EAP is calculated according to the real-time price.

2) In addition to the basic unit electricity cost of CBESS, _cbe , when its electricity is close to its limit, it may continue to operate resulting in increased cost. Finally, the total operation and maintenance cost of CBESS, C _o,m, is shown in the following formula

C ₁ =|P _be |·c _be

4) CBESS has the ability to mitigate the negative impact of MG on the distribution network. Therefore, a negative return line with a coefficient σ is added as a penalty to suppress the power (P _{exc_grid} ) fluctuation of the grid connection point

r _line = -σ · |P _{exc_grid} |

5) Once the performed action causes the SOC to exceed [0,1], a large penalty r _exc must be given to prevent the agent from making unreasonable decisions in subsequent learning. Finally, the immediate reward _rt is defined as

5.3 The specific process of the S3 step is:

Before each round of iteration starts, initialize uncertainty data, including the load curve of the microgrid, renewable distributed generation output, and market price signals. Specifically, the actual value of the load curve, RDG output and market electricity price can be given first, and it is assumed that the prediction errors obey a certain normal distribution, so as to characterize the uncertainty fluctuation.

5.4 The specific process of the S4 step is:

For the MG model, whose goal is to minimize the running cost under the forecast price signal, the objective function of its economic dispatch (ED) model is as follows:

where T is the planning period; cCDG z is the power generation cost of the zth CDG, ci ^es is the operating cost of the _ith microgrid energy storage; PCDG z,t is the power output of the zth CDG, and Pes i , t is the charging and discharging power of the i-th microgrid energy storage. Pb.grid t/Ps.grid t respectively represent the sales and purchase prices of the upper-level distribution network in each period, and P _t ^b.CHE /P _t ^s.CHE respectively represent the sales and purchase prices released by CBESS operators.

According to the forecast data, the microgrid adopts the mixed integer linear programming (MILP) method to obtain the transaction volume P _t ^mg.CHE /P _t ^mg.grid between the CBESS and the upper-level distribution network during this period, and publish the transaction information to the outside world; At the same time, the agent of CBESS obtains the state feature vector s _t = [t, SOC _t , ^pric _t b.pre , pric _t ^s.pre , P _t ^mg.CHE , P _t ^mg.grid ] by sensing the external environment

The specific process of 5.5S5 steps is:

Use s _t as input in the main competitive Q network to get the Q value output corresponding to all actions. Use the ε-greedy method to select a corresponding action a _t in the current Q value output, and execute the current action a _t in the state s _t ; for the ε-greedy strategy, first set the value of ε∈(0,1), then in the corresponding , greedily selects the optimal action a ^* currently regarded as the largest Q-value with probability (1-ε), while randomly exploring potential actions from all K discrete optional actions with probability ε

Among them, ε will gradually decrease ε _fin from ε _ini with the iterative process, so as to encourage more exploration in the early stage of the iteration, and focus on greedy convergence in the later stage, so that the algorithm can converge stably.

The specific process of 5.6S6 steps is:

S6: The power SOC _t of the CBESS is updated to SOC _t+1 , so as to judge whether the current iteration is in a terminated state, and calculate the immediate reward _rt after this action. The binary variable done is used as the indicator for determining the termination of the iteration, which is used as the interrupt indicator for each iteration process.

In the formula, if the state of charge exceeds the limit during the energy storage operation, the done of this iteration is equal to 1, otherwise it is 0. done=1 means to terminate and jump out of this iteration, done=0 means that the iteration is not terminated.

S7: MG conducts secondary MILP planning according to the tradable power actually fed back by CBESS, determines the transaction power between the current period and the external system, and gives the pre-trading power Pmg.CHE t+1, Pmg.grid t+1 for the next period As the perception state information of the agent in the next period; at this time, the state of the system is updated to s _t+1 ;

(j＝1,2··,m)，计算每个样本对应的当前目标Q值y_j S8: In the process of continuous iterative update, the five-tuple {s _t , at , r _t , s composed of s _t , at _t , r _t , s _t+1 and the termination judgment index done obtained in each period _{t t+1} , done} are stored in the leaf nodes of sumtree in turn. If the stored quantity reaches the maximum capacity of the leaf node, the old data will be rolled over and new data will be stored to ensure the validity of the sample. Once the number of samples reaches m, the number of training samples in the mini-batch, start to randomly sample m samples from the leaf nodes according to the priority playback mechanism

(j=1,2··,m), calculate the current target Q value y _j corresponding to each sample

For the priority playback mechanism, the more important sample data is played back at a higher frequency. Therefore, the TD error δ needs to be calculated and stored, and the sample with a larger absolute value of δ is easier to be sampled. The proportional priority strategy is adopted, which is a random adoption strategy between the pure greedy strategy and the uniform sampling strategy, that is, the extraction probability of the i-th sample is P(i) is

where α∈[0,1], is a power exponent that converts the importance of TD errors into priorities. If α=0, convert to uniform random sampling. pi is the priority of transition _i , calculated as follows

是一个小的正偏差，以确保仍然可以提取TD误差为0的一些边缘样本。上述过程会导致随机更新的期望分布发生变化，因此收敛解也随之变化。鉴于这种情况，采用重要抽样(IS)权重来校正偏差，从而得到考虑样本优先级的均方差损失函数L_i(θi)。最后通过神经网络的梯度反向传播来更新主竞争Q网络的所有参数θin

is a small positive bias to ensure that some edge samples with a TD error of 0 can still be extracted. The above process causes the expected distribution of random updates to change, and therefore the converged solution. In view of this situation, importance sampling (IS) weights are used to correct the bias, resulting in a mean squared error loss function Li ( _θi ) considering the priority of the samples. Finally, all parameters θ of the main competitive Q network are updated through the gradient back-propagation of the neural network

ω _j =(N·P(j)) ^-β /max _i ω _i

θ _i =θ _{i -1} +α▽ _{θi Li} (θ _i )

where ω _j is the IS weight of sample j; β is a hyperparameter that gradually increases to 1. Figure 3 summarizes the structure of the priority experience playback algorithm.

S9: After the Q network is updated, recalculate and update the priority p _i of the data stored in the sumtree, and periodically copy the parameters of the main competing Q network to the target Q network, and set the current state s=s _t+1 at the same time, if s is terminated In the state, the iteration of the current round is completed, or when the iteration round number T is reached, all the iterations are ended and returned to S3 for looping, otherwise, go to step S5 to continue the iteration.

The above are only preferred embodiments of the present invention, and it should be understood that the present invention is not limited to the form disclosed herein, should not be construed as an exclusion of other embodiments, but may be used in various other combinations, modifications and environments, and Modifications can be made within the scope of the concepts described herein, from the above teachings or from skill or knowledge in the relevant field. However, modifications and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all fall within the protection scope of the appended claims of the present invention.

Claims (8)

1. the intelligent online control method of grid-connected shared energy storage system, is characterized in that, comprises the following steps: Step 1, build two multi-hidden layer competitive Q network models, the main competitive Q network and the target competitive Q network, whose input is the feature vector s _t of the observation state, and the output corresponds to the action value of at _t in each action set A Q(s _t , at _t ); Step 2, establish the Markov decision process of CBESS, map its charging and discharging behavior into a reinforcement learning process based on iterative update of action value; determine the characteristics of the environment state and the immediate reward function; Step 3: Enter E rounds of iterative learning, and each round starts to re-initialize the load curve of MG and the output of RDG, market price and SOC of shared energy storage; Step 4, the MG performs the first plan scheduling in the round, and obtains the first state vector s _t obtained by the agent perception environment with the pre-transaction volume CBESS of the external system; Step 5: Use s _t as the input in the main competitive Q network to obtain the Q value output corresponding to all actions, and use the ε-greedy method to select an optimal estimated Q value in the current Q value output to determine its corresponding action a. _t and execute; Step 6: Update the remaining power SOC _t of CBESS to SOC _t+1 , judge whether SOC _t+1 exceeds the range of [0,1] to determine whether it exceeds the limit, and calculate the termination judgment index done _t of this round of iterations, Simultaneously calculate the immediate reward _rt after this action; Step 7: MG performs secondary planning for the current period according to the tradable power actually fed back by CBESS, determines the transaction power with the external system, and gives the pre-trading power Pmg.CHE t+1, Pmg.grid t+ for the next period. 1, as the perception state information of the agent in the next period; and update the state of the system to s _t+1 ; Step 8: Calculate the priority values of s _t , at , r _t , and s _{t +1} , and store them and done _t indicators in the leaf nodes of the sumtree in turn; if the number of stored data reaches the preset small batch When sampling capacity m, randomly sample m samples from it, calculate the current target Q value and its error, and update all hyperparameters of the main competitive Q network through gradient backpropagation; Step 9, after the Q network is updated, recalculate and update the priority p _i of the stored data in the sumtree, copy the parameters of the main competition Q network to the target Q network, and make the current state s=s _t+1 simultaneously; if s is the termination state Or when the number of iteration rounds T is reached, this round of iteration is completed, and go back to step 3 to loop; otherwise, go to step 5 to continue the iteration. 2. The intelligent online control method for a grid-connected shared energy storage system according to claim 1, wherein the main competition Q network is a state value sublayer with a single neuron and a network with K neurons. The multi-hidden layer main competition Q network architecture of the action advantage sublayer, the activation function selects the ReLu function to speed up the convergence process; the normal initialization inter-layer weight ω, the initialization bias b are constants tending to 0; the sequence number, CBESS State of charge, market price, MG and CBESS/superior distribution network's pre-transaction electricity constitute state feature vector s _t as network input, output optimal discrete charge and discharge action value Q _t , and finally perform network training by preferentially replaying data to iteratively converge. 3. The intelligent online control method of the grid-connected shared energy storage system according to claim 1, wherein the action set A is: Divide the action space of CBESS into K discrete charge and discharge options P _be ^(k) , and uniformly discretize the action space A

where A is the set of all possible actions; P _be ^(k) represents the k-th charge/discharge action in the uniform discrete action space of CBESS. 4. The intelligent online control method of grid-connected shared energy storage system according to claim 1, characterized in that, in the described Markov decision-making process of establishing CBESS, the charging and discharging behavior of CBESS is mapped to iteration based on action value An updated reinforcement learning process, specifically: The remaining power of CBESS changes continuously during the charging and discharging process, and its variation is related to the charging and discharging power and self-discharge in this period; the recursive relationship of energy storage charging is SoC(t)=(1-σ _sdr )·SoC( t-1)+P _be ·(1-L _c )Δt/E _cap The energy storage discharge process is expressed as follows SoC(t)=(1-σ _sdr )·SoC(t-1)-P _be Δt/[E _cap ·(1-L _dc )] In the formula: SoC(t) is the remaining power of CBESS in period t; _Pbe (t) is the charge and discharge power of CBESS in period t; σ _sdr is the self-discharge rate of energy storage medium; L _c and L _dc are CBESS respectively Δt is the duration of each calculation window; The maximum allowable charge and discharge power of CBESS at time t is determined by its own charge and discharge characteristics and the remaining state of charge at time t, and the constraints are satisfied during operation: SoC _min ≤SoC(t)≤SoC _max where: SoC _max and SoC _min are the upper and lower limits of CBESS state-of-charge constraints, respectively; The environmental state characteristics are: Define the environmental state feature vector perceived by CBESS at time t as s _t as

In the formula, t is the sequence number;

Respectively represent the predicted sales and purchase prices of the upper-level power grid at time period t,

Represent the pre-transaction electricity between the microgrid and the CBESS and the upper-level grid, respectively; 1) The instant reward function is as follows: CBESS obtains energy arbitrage profits by charging during off-peak hours and then discharging during peak hours; after determining the actual transaction power with the microgrid and the upper-level power grid respectively, the reward income r is calculated according to the real-time price. _EAP ; The total cost of operation and maintenance of CBESS, C _o,m, is shown in the following formula C ₁ =|P _be |·c _be

A negative return line with a coefficient σ is added as a penalty to suppress the fluctuation of the power P _{exc_grid} at the grid connection point r _line = -σ · |P _{exc_grid} | If the performed action causes the SOC to exceed [0,1], a large penalty r _exc is given to prevent the agent from making unreasonable decisions in subsequent learning; the immediate reward r _t is:

5 . The intelligent online control method for a grid-connected shared energy storage system according to claim 1 , wherein the MG executes the first planned scheduling in a round, and obtains the agent of the pre-transaction volume CBESS with the external system. 6 . The first state vector s _t obtained by perceiving the environment includes the following process: For the MG model, the goal is to minimize the operating cost under the predicted price signal, and the objective function of the economic dispatch model is as follows:

In the formula, T is the planning period;

is the power generation cost of the zth CDG, and ^{ci es} is the operating cost of the _ith microgrid energy storage;

is the power output of the zth CDG,

is the charging and discharging power of the i-th microgrid energy storage;

represent the electricity sales and purchase prices of the upper-level distribution network in each period, respectively,

represent the electricity sales and purchase prices issued by the CBESS operator respectively; The microgrid adopts the mixed integer linear programming method according to the forecast data to obtain the transaction amount of electricity between the time period and the CBESS and the upper-level distribution network.

Publish transaction information to the outside world; the agent of CBESS obtains the state feature vector by sensing the external environment

6 . The intelligent online control method for a grid-connected shared energy storage system according to claim 1 , wherein in the main competitive Q network, s _t is used as the input to obtain the Q value output corresponding to all actions. 7 . , using the ε-greedy method to select an optimal estimated Q value in the output of the current Q value to determine its corresponding action a _t and execute it, including the following processes: In the main competitive Q network, use s _t as the input to obtain the Q value output corresponding to all actions; use the ε greedy method to select a corresponding action a _t in the current Q value output, and execute the current action a _t in the state _st ; for The ε-greedy strategy, first by setting the value of ε∈(0,1), then in the corresponding action, the optimal action a ^* currently regarded as the maximum Q value is greedily selected with probability (1-ε), while Randomly explore potential actions from among all K discrete optional actions with probability ε:

where ε will gradually decrease from ε _ini ε _fin over the iterative process. 7. The intelligent online control method of a grid-connected shared energy storage system according to claim 1, wherein the remaining power SoC _t of the CBESS is updated to SoC _t+1 , and it is judged that SoC _t+1 exceeds [ 0,1] range to determine whether it exceeds the limit, and calculate the termination judgment index done _t of this round of iterations, and calculate the immediate reward r _t after this action, which includes the following process: The power SoC _t of CBESS is updated to SoC _t+1 to judge whether this iteration is in the termination state, and calculate the immediate reward r _t after this action; the binary variable done is used as the indicator for judging the termination of the iteration, which is used as the interruption indicator of each iteration process

In the formula, if the state of charge exceeds the limit during the energy storage operation, the done of this iteration is equal to 1, otherwise it is 0; done=1 means termination and jumps out of this iteration, done=0 means the iteration is not terminated. 8 . The intelligent online control method for a grid-connected shared energy storage system according to claim 1 , wherein the calculation of s _t , at , r _t , and s _{t +1} in the step 8. 9 . , and store it and done _t indicators in the leaf nodes of sumtree in turn; if the number of stored data reaches the preset small batch sampling capacity m, randomly sample m samples from it, and calculate the current target Q value and its error, and update all hyperparameters of the main competitive Q network through gradient backpropagation, where the current target Q value y _j is:

The proportional prioritization strategy is adopted, that is, the P(i) of the extraction probability of the i-th sample is:

where α∈[0,1] is the power exponent that converts the importance of TD error into priority; if α=0, it is converted to uniform random sampling; p _i is the priority of conversion i, calculated as follows :

in

is a positive deviation; The important sampling weight is used to correct the deviation, so as to obtain the mean square error loss function Li ( _θi ) considering the sample priority, and finally all parameters θ of the main competitive Q network are updated through the gradient back-propagation of the neural network:

ω _j =(N·P(j)) ^-β /max ω _i

where ω _j is the IS weight of sample j; β is a hyperparameter gradually increasing to 1.

Images