JP2021034050A - Auv action plan and operation control method based on reinforcement learning - Google Patents

Abstract

To provide an AUV action plan and operation control method, which permits action control of an AUV by executing the AUV action plan if a state of emergency occurs when a structure of three layers of a task of an autonomous type unmanned diving machine, that is, a task layer, an action layer and an action layer is defined.SOLUTION: A control command generated to complete an action planned by a robot in the water is defined as an operation. An AUV, when executing a tunnel detection task, executes an action plan in real time using a deep reinforcement learning DQN algorithm, prepares an action network for corresponding deep learning, thereby executing a tunnel detection task plan. Training of an action network of the AUV is performed according to a DDPG method and mapping from a force to a state is obtained as the AUV as an environmental model, thereby achieving operation control of the AUV.SELECTED DRAWING: Figure 1

Description

The present invention belongs to the technical field of underwater robots, and specifically relates to AUV action planning and motion control methods.

The 21st century is a century in which the use of the ocean is advancing, and efforts in the marine industry are recognized by countries around the world. China has also announced and implemented an important maritime strategy, and since China is currently in a fast-developing stage and has a large population with limited land resources, marine resources are important to support sustainable development. It is a resource storage space. The development and exploration of marine resources is an important premise and foundation for implementing marine strategies, and as a major underwater technology and equipment, the Autonomous Underwater Vehicle (AUV) is for civilian use in the ocean. It has become a practical and effective development tool in research fields such as military and science, and is an important means of marine development and exploration. The application of autonomous underwater vehicles and the research and development / upgrade of technology can be expected to attract attention in the future, and will be an important means for countries around the world to gain a leading position in the marine field. AUV research requires the application of various high-end technologies, including many technologies such as navigation / positioning, underwater target detection, communication technology, intelligent control, simulation technology, energy system technology and planning technology.

Planning and control technology is one of the key elements for achieving AUV intelligence, and is the premise and foundation for AUVs to make autonomous decisions and complete work tasks. The underwater environment is complex, dynamic, unstructured and uncertain, making it difficult to obtain underwater environment information, and therefore AUVs face unexpected emergencies when working underwater. It is inevitable to do. Due to restrictions on underwater communication means, the AUV has no choice but to rely on its own decisions to deal with emergencies, which requires the AUV to modify its original planning instructions and replan in response to environmental emergencies. There is. The present invention focuses on AUV planning technology in a complex environment, makes a pressurized water supply tunnel a complex underwater environment, and uses a tunnel detection task as a representative example of the application background, and an AUV action plan based on reinforcement learning. And the control method is proposed.

Detection of pressurized water tunnels is one of the important items of irrigation work management, and pressurized water tunnels are underground water channels for connecting irrigation projects such as hydroelectric power plants. Currently, most hazards in small and medium-sized reservoirs are due to the failure to detect the risks of water tunnels in a timely manner, and long-term operation of the tunnels results in defects and problems such as deposition, corrosion, water leaks and obstacles. In particular during the flood season, water tunnels are prone to pipeline aging problems and have a direct impact on the safe operation of underwater projects, so the tunnels should be inspected regularly for the status of underwater projects. It has been emphasized to grasp. However, due to problems such as the small diameter of some tunnels, the high flow velocity during the flood season, and the complicated underwater construction environment, workers can enter the tunnel for inspection. , Other detectors must be used on behalf of the inspector to complete the inspection task.

An autonomous underwater vehicle (AUV) is a good platform for mounting underwater detectors, capable of autonomously and safely performing target tasks underwater for long periods of time, and is powerful in the complex water environment of tunnels. It is flexible and state-sustainable, equipped with underwater detectors and sensors to meet the detection needs, can autonomously complete tunnel detection tasks, and functions as the main means for tunnel detection. It is expected. The present invention designs an action plan and a control method for an AUV tunnel detection task based on a reinforcement learning algorithm, enhances the environmental adaptability of the AUV, and improves the ability to determine the AUV in an emergency.

The present invention is accurate for problems where the intelligent level of underwater robots is insufficient and relies too much on artificial experience when performing complex tasks, and for control methods designed based on intelligent algorithms in conventional underwater robots. It solves the problem that the training experience is limited and it is difficult to apply it to the real environment due to the need for an environmental model.

第２項ｒ₁₂はＡＵＶの向首角の変化を考慮し、向首角がターゲットに近いほど、ターゲット値の報酬値が大きく、現在のＡＵＶ向首とターゲット向首との夾角αが
α＝θ−βであり、
αの絶対値が小さいほど、取得する報酬値が大きく、具体的には、
ｒ₁₂＝ｋ_Aｃｏｓ（α）
式中、ｋ_Aはターゲットへの移動過程に対応する報酬係数であり、
総報酬値は２項を加重したものであり、
ｒ₁＝ｋ₁₁ｒ₁₁＋ｋ₁₂ｒ₁₂
式中、ｋ₁₁、ｋ₁₂はそれぞれ加重値であり、
前記行動のうち壁追跡過程においては、以下のステップを含み、
ＡＵＶ壁追跡行動は、ＡＵＶと壁の距離及び相対角度の情報を考慮し、ＡＵＶは、一方の側に配置された前後にある２つのレンジングソナーを通じて壁からのＡＵＶの距離ｘ₄とｘ₅を取得し、
方位磁針で現在のＡＵＶ向首角θを取得して、現在の壁角度θ_wallを推定し、

式中、ｌ_AUVは前後にある２つのソナーの間の距離であり、壁追跡行動の環境状態の特徴入力はｘ₁、ｘ₂、ｘ₃、ｘ₄、ｘ₅、θ、θ_wall及びターゲット追跡距離ｌ_goalに設定され、ここで、ｘ₁〜ｘ₅はそれぞれ３つの前方ソナーと一側に設けられた前後ソナーにより測定されたデータであり、特徴入力量は８個であり、ＡＵＶと壁の間の状態関係を完全に記述することができ、距離閾値を設定してソナーデータについて判断を行い、トレーニング中に閾値を超えると、現在のトレーニング周期を終了し、
報酬関数は、ＡＵＶができるだけ壁に平行し、壁とは一定の距離を保持するようにし、仮想ターゲットポイントに基づく壁追跡行動の報酬信号が主に４項に設定され、一般的な壁追跡行動の報酬信号が主に２項に設定され、
第１項は、式（６）のようにＡＵＶと現在の壁がなす角度を考慮し、ＡＵＶと壁の角度が増大し閾値を超えると、負の報酬値を取得し、ＡＵＶと壁の角度が減少すると、正の報酬値を取得し、

第２項は、式（７）のようにＡＵＶの前後両端と壁の距離を考慮し、ＡＵＶと壁の距離と予め設定された値との差が減少すると、正の報酬を取得し、この差が増大すると負の報酬を取得し、追跡距離が予め設定された値の±０．２ｍの範囲にあることができ、追跡範囲内のこの項の報酬値が０である場合、この場所と壁の距離値は、同一側面にある２つのレンジングソナーによるデータの平均値であり、

一般的な壁追跡行動の総報酬ｒは２項の報酬を加重したものであり、
ｒ₂＝ｋ₂₁ｒ₂₁＋ｋ₂₂ｒ₂₂
式中、ｋ₂₁、ｋ₂₂はそれぞれ加重値であり、
仮想ターゲットポイントに基づく追跡では、この仮想ターゲットポイントは外直角と内直角の壁により作成された仮想ターゲットポイントであり、環境が外直角である場合、前側のソナーが障害物を検知していないときに入力が最大検出距離であるので、仮想壁が構築され、仮想ターゲットポイントが追加され、環境が内直角である場合、前方ソナーが壁を検知したとき、ＡＵＶが対向する現在のターゲット壁の他方の側で仮想ターゲットポイントが構築され、
仮想ターゲットポイントに基づく報酬関数の構築は、以下のとおりであり、

ｒ₂₄＝ｋ_Bｃｏｓ（α）
式中、ｋ_Bは壁追跡過程に対応する報酬係数であり、
仮想ターゲットポイントに基づく追跡行動の総報酬ｒは４項の報酬を加重したものであり、
ｒ₂＝ｋ₂₁ｒ₂₁＋ｋ₂₂ｒ₂₂＋ｋ₂₃ｒ₂₃＋ｋ₂₄ｒ₂₄
式中、ｋ₂₃、ｋ₂₄はそれぞれ加重値であり、
ＡＵＶが次の部分の壁を追跡するまで徐々に調整したとき、たとえば、外直角環境におけるレンジングソナーが再度ターゲット壁を検知したか、内直角環境における前方ソナーがさらに前方の壁を検知しない場合、仮想ターゲットポイントを削除し、一般的な壁追跡行動ネットワークを呼び出し、
前記行動のうち障害物回避過程においては、以下のステップを含み、
ＡＵＶ障害物回避行動のニーズについて、環境状態の特徴入力は３つの前方ソナーと両側のそれぞれに設けられたフロントソナーのデータを含み、ＡＵＶは、障害物を回避しながらターゲットポイントの方向へ近づき、特徴入力はＡＵＶの現在の位置座標（ｘ_AUV,ｙ_AUV）、ターゲットポイント位置座標（ｘ_goal,ｙ_goal）、現在の向首角θ及びターゲット向首角βの計１１次元の入力をさらに含み、
報酬関数については、報酬信号が３項に分けられ、第１項は障害物に対するＡＵＶ距離に基づいて得られた報酬値ｒ₃₁であり、ＡＵＶが障害物に近づくと、負の報酬の警告を取得し、ＡＵＶが障害物から離間すると、正の報酬を取得し、ＡＵＶが障害物から離間して航行するように促し、障害物と衝突すると報酬値−１を取得し、現在のトレーニング周期を終了し、

第２項は、現在のＡＵＶとターゲットポイントの距離に基づいて生じる報酬値ｒ₃₂であり、ＡＵＶが障害物を回避しながらターゲットポイントへ航行するように促し、このため、ＡＵＶがターゲットポイントから離間すると、負の報酬を取得し、ターゲットポイントに近づくと正の報酬を取得し、ＡＵＶターゲットポイントに到着すると、正の報酬値１．０を取得し、トレーニング周期を終了し、

第３項は、ＡＵＶと現在のターゲットがなす角度αに基づいて生じる報酬ｒ₃₃であり、同様にＡＵＶがターゲットポイントの方向へ航行するように促すが、この項の報酬は、主に、現在のターゲット向首に近くなるように向首角を調整することをＡＵＶに学習させ、経路の長さを減らすようにするためであり、
ｒ₃₃＝ｋ_cｃｏｓ（α）
式中、ｋ_Cは壁障害物の回避過程に対応する報酬係数であり、
最後の総報酬信号はこの３項の報酬値を加重したものに等しく、
ｒ₃＝ｋ₃₁ｒ₃₁＋ｋ₃₂ｒ₃₂＋ｋ₃₃ｒ₃₃
式中、ｋ₃₁〜ｋ₃₃はそれぞれ加重値であり、
強化学習は、動作から環境へのマッピングをトレーニングするものであり、ロボットを環境として、ＤＤＰＧトレーニングを通じて力とトルクを得て水中ロボットに作用させ、ＡＵＶモデルを用いて計算することによりロボットの速度と角速度を得て、速度、角速度とターゲット速度、ターゲット角速度との誤差を利用して報酬値ｒ₄＝−｜△ｖ＋△Ψ｜を設計し、ここで△ｖは速度誤差であり、△Ψは向首誤差であり、
また、トレーニング中のＡＵＶモデルにランダム干渉力を追加することで、ＤＤＰＧに基づく制御システムをトレーニングにより得て、制御システムのトレーニングが完了した後、ロボットの現在の位置及びターゲット経路から、経路追跡戦略に従ってターゲット命令を得て、ＤＤＰＧ制御システムを用いてロボットを計画命令に従うように制御する。 AUV action plan and motion control method based on reinforcement learning.
Underwater robot tunnel detection is defined as a total task, or task, and actions to complete the task include moving to a target, wall tracking, and obstacle avoidance, completing the robot-planned action underwater. And the steps to define the concrete control instructions that occur for
When the AUV performs the tunnel detection task, it uses the deep reinforcement learning DQN algorithm to perform action planning in real time based on the underwater environment to be detected, that is, it builds an action planning architecture based on multi-action network calls. Steps to define the input and output behaviors of the environmental state characteristics of the three behaviors according to the needs of the task, build the corresponding deep learning behavioral network, and design the reward function.
The steps that the planning system completes the tunnel detection task by calling the trained behavior network,
Includes steps in which the control system completes the planned action by calling the trained operating network.
In the AUV action planning and motion control method based on the reinforcement learning, the process of constructing the corresponding deep learning action network and designing the reward function includes the following steps.
The tunnel detection task is broken down into action sequences, multiple feasible route points are planned based on prior environmental information in global route planning, the AUV departs from the placement position and arrives at each route point in sequence.
Since the route point is a global plan under a known environment, during the travel process, the AUV calls the obstacle avoidance action to safely arrive at the route point based on the real-time environmental state, and the AUV is the main tunnel detection task. Call wall tracking action to complete the task according to a given detection target,
The decision module includes global data, decision system, action library and evaluation system, and task information, situation information and planning knowledge are stored in the global data. The decision system is a self-learning planning system combined with the DQN algorithm. , Trained, extract trained network parameters from the behavior library prior to performing the planning task, then plan the current behavior behavior using the current environmental state information as input, and the evaluation system is an enhanced learning algorithm. It is a reward function system of the above, and each time the AUV plans and executes one action action plan, it provides a reward based on the state environment and task information, and all the data is stored in the global database. In the process of moving to the target, the following steps are included.
The movement action to the target point is to navigate to the target point while adjusting the head angle when the AUV does not detect an obstacle, and the relationship between the position and angle of the AUV and the target point is mainly used as the feature input amount. In consideration, specifically, the current AUV position coordinates (x _AUV , y _AUV ), the target point coordinates (x _goal , y _goal ), the current heading angle θ, and the target heading angle β are input in a total of 6 dimensions. Among them, the target heading angle β is the heading angle when the AUV is navigating to the target.
Regarding the reward function, when the AUV navigates to the target point in an obstacle-free environment due to the movement action to the target, the reward function is set to the second term.
Item 1 r ₁₁ considers the change in the distance between the AUV and the target point.

The second term r ₁₂ considers the change in the AUV heading angle, and the closer the heading angle is to the target, the larger the reward value of the target value, and the current AUV heading angle α between the target heading angle is α = θ-β,
The smaller the absolute value of α, the larger the reward value to be acquired.
r ₁₂ = k _A cos (α)
In the formula, k _A is the reward coefficient corresponding to the process of moving to the target.
The total reward value is the weight of two terms,
r ₁ = k ₁₁ r ₁₁ + k ₁₂ r ₁₂
In the formula, k ₁₁ and k ₁₂ are weighted values, respectively.
The wall tracking process of the above actions includes the following steps:
The AUV wall tracking behavior takes into account information on the distance and relative angle between the AUV and the wall, and the AUV measures the distance x ₄ and x ₅ of the AUV from the wall through two front and rear ranging sonars located on one side. Acquired,
Obtain the current AUV head angle θ with the compass, estimate the _{current wall angle θ wall, and}

In the equation, l _AUV is the distance between the two sonars in front and behind, and the environmental state feature inputs of the wall tracking behavior are x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , θ, θ _wall and target. The tracking distance is set to _{l goal , where x 1 to x 5} are data measured by three front sonars and front and rear sonars provided on one side, respectively, and the feature input amount is eight, which is AUV. You can fully describe the state relationship between the walls, set a distance threshold to make decisions about sonar data, and if the threshold is exceeded during training, the current training cycle ends.
The reward function is to keep the AUV as parallel to the wall as possible and keep a certain distance from the wall, and the reward signal of the wall tracking action based on the virtual target point is mainly set in the 4th term, and the general wall tracking action. Reward signal is mainly set to 2 items,
The first term considers the angle formed by the AUV and the current wall as in equation (6), and when the angle between the AUV and the wall increases and exceeds the threshold value, a negative reward value is obtained and the angle between the AUV and the wall is obtained. When is reduced, you get a positive reward value,

The second term considers the distance between the front and rear ends of the AUV and the wall as in equation (7), and when the difference between the distance between the AUV and the wall and the preset value decreases, a positive reward is obtained. If the difference increases, you will get a negative reward, the tracking distance can be in the range of ± 0.2m of the preset value, and if the reward value of this item in the tracking range is 0, then with this location The wall distance value is the average of the data from two ranging sonars on the same side.

The total reward r for general wall tracking behavior is the weight of the reward in the second term.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ r ₂₂
In the formula, k ₂₁ and k ₂₂ are weighted values, respectively.
In tracking based on virtual target points, this virtual target point is a virtual target point created by a wall with an outer right angle and an inner right angle, and if the environment is an outer right angle, when the front sonar is not detecting an obstacle. Since the input to is the maximum detection distance, a virtual wall is built, a virtual target point is added, and if the environment is at right angles, when the forward sonar detects the wall, the AUV faces the other of the current target walls. A virtual target point is built on the side of
The construction of the reward function based on the virtual target points is as follows,

r ₂₄ = k _B cos (α)
In the formula, k _B is the reward factor corresponding to the wall tracking process.
The total reward r for tracking behavior based on virtual target points is the weight of the reward in item 4.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ r ₂₂ + k ₂₃ r ₂₃ + k ₂₄ r ₂₄
In the formula, k ₂₃ and k ₂₄ are weighted values, respectively.
When the AUV is gradually adjusted until it tracks the next part of the wall, for example, if the ranging sonar in the outer right angle environment detects the target wall again, or if the anterior sonar in the inner right angle environment does not detect the further anterior wall. Delete the virtual target point, call the general wall tracking behavior network,
Among the above actions, the obstacle avoidance process includes the following steps.
For the needs of AUV obstacle avoidance behavior, the environmental state feature inputs include data for the three front sonars and front sonars on each side, and the AUV approaches the target point while avoiding obstacles. The feature input further includes a total of 11-dimensional inputs of the current position coordinates (x _AUV , y _AUV ) of the AUV, the target point position coordinates (x _goal , y _goal ), the current heading angle θ, and the target heading angle β. ,
Regarding the reward function, the reward signal is divided into three terms, the first term is the reward value r ₃₁ obtained based on the AUV distance to the obstacle, and when the AUV approaches the obstacle, a negative reward warning is given. Obtain and get a positive reward when the AUV moves away from the obstacle, encourage the AUV to navigate away from the obstacle, get a reward value of -1 when colliding with the obstacle, and set the current training cycle. Finished

_{The second term is the reward value r 32} generated based on the current distance between the AUV and the target point, which encourages the AUV to navigate to the target point while avoiding obstacles, thus causing the AUV to move away from the target point. Then, a negative reward is obtained, a positive reward is obtained when approaching the target point, a positive reward value of 1.0 is obtained when the AUV target point is reached, and the training cycle is completed.

_{The third term is the reward r 33} generated based on the angle α between the AUV and the current target, which also prompts the AUV to navigate in the direction of the target point, but the reward in this section is mainly the present. This is to train the AUV to adjust the heading angle so that it is closer to the target heading, and to reduce the length of the path.
r ₃₃ = k _c cos (α)
In the formula, k _C is the reward coefficient corresponding to the wall obstacle avoidance process.
The final total reward signal is equal to the weight of the reward values in these three terms,
r ₃ = k ₃₁ r ₃₁ + k ₃₂ r ₃₂ + k ₃₃ r ₃₃
In the formula, k _{31 to k 33} are weighted values, respectively.
Reinforcement learning trains the mapping from motion to environment. Using the robot as the environment, the force and torque are obtained through DDPG training to act on the underwater robot, and the speed of the robot is calculated by using the AUV model. _{Obtain the angular velocity and design the reward value r 4} =-| △ v + △ Ψ | using the error between the velocity, the angular velocity and the target velocity, and the target angular velocity, where Δv is the velocity error and ΔΨ is It is a heading error,
In addition, by adding random interference force to the AUV model during training, a control system based on DDPG is obtained by training, and after the training of the control system is completed, the route tracking strategy is performed from the current position and target route of the robot. The target command is obtained according to the above, and the robot is controlled to follow the planned command by using the DDPG control system.

The beneficial effects of the present invention are as follows.
1. The three-layer planning system designed by the present invention decomposes the total task into sub-actions of moving to the target and avoiding obstacles, designs an environmental state model and a reward function, and optimizes the space during the operation. It reduces dimensions, allowing you to plan safe, collision-free routes even in complex environmental models, solving the problem of "curse of dimensionality".
In addition, the present invention has a high level of intelligence, the planning does not need to rely on manual programming, and robot control can be realized without relying on artificial experience.
2. The present invention solves the problem of detection in continuous environmental conditions by applying a deep reinforcement learning algorithm to an action planning system and extracting high-dimensional data features via a neural network, and also provides reinforcement learning. Is used to make an action decision plan. Three typical actions are defined, moving to the target point, wall tracking, and obstacle avoidance, depending on the needs of the tunnel detection task, and a working network is built for each action, with corresponding environmental state variables and reward functions. Designed, for wall corner problems, a virtual target point-based tracking method is proposed. Each action has reached its corresponding goal and completes the tunnel detection task by calling each action network, which makes the algorithm more stable and more generalizable.
3. Since the present invention trains the mapping relationship from force to velocity using the kinematic model of AUV as an environment, the control method of the present invention does not require an accurate environmental model and the training experience is limited. It solves the problem that it is difficult to apply in the real environment, has universal adaptability compared to the research of other intelligent control algorithms, and can be applied to various tasks after successful training only once. ..

It is a schematic diagram which divided the task of an autonomous underwater vehicle into three layers. It is an exploded schematic diagram of a task. It is a schematic diagram of a wall tracking behavior. It is a schematic diagram of the outer right angle wall environment. It is a schematic diagram of the inner right angle wall environment. It is a schematic diagram of an obstacle avoidance behavior. It is a layout drawing of AUV sonar.

Embodiment 1
This embodiment is an AUV action plan and motion control method based on reinforcement learning.

The present invention defines a three-layer structure of tasks of an autonomous underwater vehicle, that is, a task layer, an action layer, and an action layer, and an AUV action plan is executed in the event of an emergency, and the Deep Deterministic Policy The behavior of the AUV is controlled by the DDPG) controller.

The realization process includes the following three parts.
(1) Hierarchical design of tasks for autonomous underwater vehicle (2) Construction of action planning system (3) Design based on DDPG control algorithm.

Further, the process of (1) above is as follows.
To layer tunnel detection tasks by underwater robots, define the concept of tunnel detection tasks, actions and actions by autonomous underwater vehicle, that is, define tunnel detection by autonomous underwater vehicle as total task, and total. To complete a task, we define three typical actions: moving to a target, tracking walls and avoiding obstacles, and the specific controls that occur as the robot navigates underwater to complete the planned action. Instructions are defined as actions, such as n-degree left turn, n-degree right turn, and n-knot speed advancement.

As shown in FIG. 1, the architecture of the action planning system of the autonomous underwater vehicle is divided into three layers: a total task layer, an action layer, and an action layer. This model is a bottom-to-top layered framework where the working layer can be thought of as the process by which the AUV interacts with the environment, where the AUV performs actions and acts on the environment and is a planning system. Updates global planning knowledge by acquiring real-time environment and self-state data through this layer and learning and training according to the previous learning experience of the planning system. The historical experience information of the environmental state data in the training sample library is compared and analyzed with the current environmental state, and then the comparison result and the planning knowledge update data are fed back to the layer task layer, and the total task layer is mainly the current one. It is the upper layer of the planning system that analyzes the environmental state, outputs the planning result according to a specific internal strategy, and sends it to the action layer in the form of the action action sequence, that is, plans the action sequence according to the current environmental state data. The action layer is an intermediate layer that selects actions according to a specific strategy based on the planning results of the upper layer by the total task layer, mainly considering the local environment state information acquired by the current operation layer. As described above, the total task layer provides the plan result of the upper layer based on the environmental state data, the action layer selects and executes the action based on the plan result of the upper layer, and the action layer The AUV performs basic actions according to the action strategy and plays a role of detecting changes in the environmental state, and the three-layer framework constitutes the decision model of the bottom-up plan.

The process of (2) above is as follows.
When performing a tunnel detection task, the AUV sequentially arrives at the critical route points specified by the global route planning, depending on the needs of the task. However, in the actual work process, there is unknown environmental information such as sudden obstacles and changes in the tunnel wall environment due to damage to the tunnel wall. It is necessary to respond in a timely manner based on the situation of. Action planning systems based on deep reinforcement learning employ a reaction-based planning architecture that allows AUVs to quickly plan actions in response to changes in the environment by building mapping relationships between environmental states and actions. It is possible to improve the ability of AUV to cope with changes in the emergency environment.

The present invention employs an AUV that detects an intelligent pressurized water supply tunnel as a research target, and this AUV detects the underwater environment using an underwater acoustic device or sensor equipped in the AUV, and uses a deep enhanced learning DQN algorithm. Use to do action planning in real time, i.e. build an action planning architecture based on multi-action network call calls, define input and output behaviors of the environmental state features of the three basic behaviors according to the needs of the task. The corresponding deep learning motion network is constructed, the reward function is designed, and the wall tracking behavior proposes a tracking method based on a virtual target point for the problem of the corner of the wall.

Regarding the problem of action layer planning, the present invention proposes three typical actions of moving to a target, wall tracking action, and obstacle avoidance action as typical examples of the application background of tunnel detection, and proposes three typical actions of the bottom layer. The basic behavior is defined, the behavior network is designed, and the planning system completes the tunnel detection task by calling the trained behavior network. For the tunnel detection task, this task can be broken down into a series of action sequences, the global route planning plans multiple viable route points based on prior environmental information, and the AUV is the placement position, as shown in FIG. Depart from and arrive at each route point in sequence.

The voyage task is that the AUV arrives at each critical point from the start point of each route, and different speed constraints can be set for each voyage task. Since the route point is a global plan under known environment, each track is not unique because the AUV calls the obstacle avoidance action according to the real-time environmental condition and safely arrives at the route point during the voyage. The tunnel detection task starts at route point 3 and ends at route point 4, where the AUV primarily calls wall tracking actions to complete the task according to a predetermined detection target.

In addition, detection modules (including sonar) within the architecture acquire real-time AUV status information and environmental information by acquiring AUV sensor data and analyzing the data according to behavioral needs. The decision module is the core of the planning system, including global data, decision systems, action libraries and evaluation systems. Task information, situation information, planning knowledge, etc. are stored in the global data, and the decision system is also a self-learning planning system combined with the DQN algorithm. The trained network parameters are extracted from the behavior database prior to execution, then the current behavioral behavior is planned using the current environmental state information as input, and the evaluation system is the reward function system of the enhanced learning algorithm, AUV. As they plan and execute action actions, they provide rewards based on state environment and task information, and all data is stored in a global database.

2.1) Move to target The AUV must reach a pre-globally planned target point during the process of performing the tunnel detection task, and move to the target point to minimize the route. The behavior is to navigate to the target point while adjusting the head angle when the AUV is not detecting an obstacle, and therefore control the real-time heading of the AUV in the process of moving to the target as close to the target direction as possible. There is a need to. As shown in FIG. 2, the feature input amount mainly considers the relationship between the AUV and the position and angle of the target point according to the needs of the movement behavior to the target, and specifically, the current AUV position coordinates (x). _AUV , y _AUV ), target point coordinates (x _goal , y _goal ), current heading angle θ, and target heading angle β, for a total of 6-dimensional inputs. The target heading angle β is the heading angle when the AUV is navigating to the target.

第２項ｒ₁₂はＡＵＶの向首角の変化を考慮し、ＡＵＶがターゲット向首に調整して航行するように促し、首角がターゲットに近いほど、ターゲット値の報酬値が大きく、現在のＡＵＶ向首とターゲット向首との夾角αが、
α＝θ−β （２）であり、
αの絶対値が小さいほど、取得する報酬値が大きく、具体的には、
ｒ₁₂＝ｋ_Aｃｏｓ（α） （３）
式中、ｋ_Aはターゲットへの移動過程に対応する報酬係数であり、
総報酬値は２項を加重したものであり、
ｒ₁＝ｋ₁₁ｒ₁₁＋ｋ₁₂ｒ₁₂ （４）
式中、ｋ₁₁、ｋ₁₂はそれぞれ加重値である。 2.1.1) Design of reward function The movement behavior to the target is mainly driven so that the AUV navigates to the target point in an obstacle-free environment, and therefore the specific reward function is set in the second term. , Paragraph 1 r ₁₁ considers the change in the distance between the AUV and the target point, specifically,

The second term r ₁₂ In view of the change of direction neck angle AUV, AUV is prompted to sail to adjust the target Kokubi, as the neck angle is close to the target, increased compensation value of the target value, the current The angle α between the AUV head and the target head is
α = θ−β (2),
The smaller the absolute value of α, the larger the reward value to be acquired.
r ₁₂ = k _A cos (α) (3)
In the formula, k _A is the reward coefficient corresponding to the process of moving to the target.
The total reward value is the weight of two terms,
r ₁ = k ₁₁ r ₁₁ + k ₁₂ r ₁₂ (4)
In the equation, k ₁₁ and k ₁₂ are weighted values, respectively.

2.2) Wall tracking Due to the long distance of most tunnels, the entire irrigation project can reach more than 10km, and once the AUV enters the tunnel entrance, manual intervention becomes difficult, which makes the AUV difficult. It is required to complete the detection task autonomously according to the tunnel environment. To avoid collisions, the AUV must be a safe distance from the wall, and the distance between the AUV and the wall directly affects the quality of image acquisition, limited by underwater light sources and visibility. Therefore, the AUV is required to have the ability to navigate along the wall while maintaining a certain distance from the wall.

2.2.1) Depending on the needs of the wall tracking function of the AUV, this action mainly considers information on the distance and relative angle between the AUV and the wall. As shown in FIG. 3, in the example where the AUV navigates by tracking the wall on the right side of itself, the AUV measures the distance x ₄ and x ₅ of the AUV from the wall through two front and rear ranging sonars placed on the right side. get.

In the present embodiment, the AUV is provided with a total of seven ranging sonars, and as shown in FIG. 7, three front sonars (1, 2, 3 in FIG. 7) are provided at the front end of the AUV. Two sonars (4, 5, and 6, 7 in FIG. 7) are provided on each side of the AUV, one on each side is provided on the front and back, and the front end is the front sonar. The rear end is called the rear sonar.

式中、ｌ_AUVは前後の２つのソナーの間の距離であり、壁追跡行動の環境状態の特徴入力はｘ₁、ｘ₂、ｘ₃、ｘ₄、ｘ₅、θ、θ_wall及びターゲット追跡距離ｌ_goalに設定され、ここで、ｘ₁〜ｘ₅はそれぞれ３つの前方ソナーと一側に設けられた前後ソナー（本実施形態では、１−５ソナーと記載する）により測定されたデータであり、特徴入力量は８個であり、前方ソナーと側面ソナーのデータを含み、前方ソナーは主に壁環境での前方の壁からの距離ｘ₁を検出し、以上の特徴変量はＡＵＶと壁の間の状態関係を完全に記述することができる。距離閾値を設定してソナーデータについて判断を行い、トレーニング中に閾値を超えると、現在のトレーニング周期を終了する。 The current AUV head angle θ is acquired with the compass, and the current wall angle θ _wall is estimated.

In the equation, l _AUV is the distance between the two sonars before and after, and the environmental state feature inputs of the wall tracking behavior are x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , θ, θ _wall and target tracking. The distance is set to _{l goal , where x 1 to x 5} are data measured by three front sonars and front and rear sonars (referred to as 1-5 sonars in this embodiment) provided on one side, respectively. Yes, the feature input amount is 8, and the data of the front sonar and the side sonar are included. The front sonar mainly _{detects the distance x 1} from the front wall in the wall environment, and the above feature variables are AUV and wall. The state relationship between them can be completely described. A distance threshold is set to make decisions about sonar data, and if the threshold is exceeded during training, the current training cycle ends.

2.2.2) Design of reward function In AUV wall tracking behavior learning, the reward function keeps the AUV as parallel to the wall as possible, keeps the angle with the wall at about 0 °, and keeps a certain distance from the wall. It is used to make it work.

In consideration of the above factors, the reward signal for the wall tracking action based on the virtual target point is mainly set in the fourth term, and the reward signal for the general wall tracking action is mainly set in the second term.

第２項は、式（７）のようにＡＵＶの前後両端と壁の距離を考慮し、ＡＵＶと壁の距離と予め設定された値との差が減小すると、正の報酬を取得し、この差が増大すると、負の報酬を取得し、追跡距離が予め設定された値の±０．２ｍの範囲にあることができ、追跡範囲内のこの項の報酬値が０である場合、この場所と壁の距離値は、同一側面にある２つのレンジングソナーデータによる平均値である。

一般的な壁追跡行動の総報酬ｒは２項の報酬を加重したものであり、
ｒ₂＝ｋ₂₁ｒ₂₁＋ｋ₂₂ｒ₂₂ （８）
式中、ｋ₂₁、ｋ₂₂はそれぞれ加重値である。 The first term considers the angle formed by the AUV and the current wall as in equation (6), and when the angle between the AUV and the wall increases and exceeds the threshold value, a negative reward value is obtained and the angle between the AUV and the wall is obtained. When is reduced, you get a positive reward value,

The second term considers the distance between the front and rear ends of the AUV and the wall as in equation (7), and when the difference between the distance between the AUV and the wall and the preset value decreases, a positive reward is obtained. If this difference increases, you will get a negative reward and the tracking distance can be in the range of ± 0.2m of the preset value, and if the reward value for this item in the tracking range is 0, then this The distance value between the location and the wall is the average value of two ranging sonar data on the same side.

The total reward r for general wall tracking behavior is the weight of the reward in the second term.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ r ₂₂ (8)
In the formula, k ₂₁ and k ₂₂ are weighted values, respectively.

2.2.3) Tracking method based on virtual target points In a general wall environment, wall tracking behavior only needs to consider the target's head angle and target's tracking distance, and movement behavior and obstacles to the target. Compared to object avoidance behavior, there is actually no guidance by the target point, so it is not possible to provide accurate planning results in special environments such as corners of walls. The wall corner problem is a major challenge in AUV wall tracking behavior, and the present invention primarily considers two types of wall corner environments: the outer right angle environment and the inner right angle environment. Due to the peculiarities of the environment in the corners of the wall, when tracking the outer right angle, the range sonar in front of the AUV cannot detect the wall, the AUV cannot adjust the head angle in a timely manner, and the target can be lost. is there. In the case of the inner wall corner, a collision occurs because the design of the base reward does not consider the obstacle in front.

環境が内直角である場合、図５に示すように、仮想壁を構築できず、ＡＵＶがタイムリーに方向を変更して前方の壁の障害物を回避することを考慮すると、前方ソナーにより壁が検知されると、ＡＵＶが対向する現在のターゲット壁の他方の側で仮想ターゲットポイントが構築され、仮想ターゲットポイントの位置はＡＵＶ位置、向首角及び安全距離Ｌ₂により決定される。

２種の環境のいずれにも安全距離Ｌ₁とＬ₂が設定され、シミュレーションテストを行った結果、その値がターゲット追跡距離程度である場合、行動計画の効果が良好である。仮想ターゲットポイントに基づく報酬関数の構築は以下のとおりである。

ｒ₂₄＝ｋ_Bｃｏｓ（α） （１４）
式中、ｋ_Bは壁追跡過程に対応する報酬係数であり、
仮想ターゲットポイントに基づく追跡行動の総報酬ｒは４項の報酬を加重したものである。
ｒ₂＝ｋ₂₁ｒ₂₁＋ｋ₂₂＋ｒ₂₂＋ｋ₂₃ｒ₂₃＋ｋ₂₄ｒ₂₄ （１５）
式中、ｋ₂₃、ｋ₂₄はそれぞれ加重値であり、
報酬係数ｋ₂₃とｋ₂₄値が大きいため、壁の隅の環境ではＡＵＶは仮想ターゲットポイントにより案内される傾向がある。 To address this issue, the present invention proposes a method of constructing virtual target points to guide AUV wall tracking. 4 and 5 show virtual target points constructed for the outer right-angled wall and the inner right-angled wall. If the environment is at right angles, the input is the maximum detection distance when the front sonar is not detecting obstacles, so the virtual wall is constructed like a dotted line, based on which virtual target points are added. .. Position of the virtual target points, AUV position is determined ranging sonar data, and the safe distance L _1.

If the environment is at right angles to the inside, as shown in FIG. 5, the virtual wall cannot be constructed, and considering that the AUV changes direction in a timely manner to avoid obstacles on the front wall, the wall is provided by the front sonar. Is detected, a virtual target point is constructed on the other side of the current target wall to which the AUV faces, and the position of the virtual target point is determined by the _{AUV position, heading angle, and safety distance L 2.}

When the safety distances L ₁ and L ₂ are set in both of the two environments and the value is about the target tracking distance as a result of the simulation test, the effect of the action plan is good. The construction of the reward function based on the virtual target points is as follows.

r ₂₄ = k _B cos (α) (14)
In the formula, k _B is the reward factor corresponding to the wall tracking process.
The total reward r of the tracking action based on the virtual target point is the weight of the reward of item 4.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ + r ₂₂ + k ₂₃ r ₂₃ + k ₂₄ r ₂₄ (15)
In the formula, k ₂₃ and k ₂₄ are weighted values, respectively.
Due to the large reward coefficients k ₂₃ and k ₂₄ , the AUV tends to be guided by virtual target points in a wall corner environment.

When the AUV is gradually adjusted until it tracks the next part of the wall, that is, if the ranging sonar in the outer right angle environment detects the target wall again, or if the forward sonar in the inner right angle environment does not detect the further forward wall. Delete the virtual target point and call the general wall tracking behavior network.

2.3) Obstacle Avoidance Obstacle avoidance behavior is a key to the action planning system, which determines the autonomous decision level of the AUV and determines whether the AUV can safely perform work tasks.

2.3.1) As shown in FIG. 6, the action planning system needs to sufficiently acquire the surrounding obstacle environmental information according to the needs of the AUV obstacle avoidance action, so that the characteristics of the environmental state are input. Contains data from three front sonars and front sonars on each side.

Since the AUV needs to approach the target point while performing obstacle avoidance and acquire the relative position information between the AUV and the target point, the feature input is the current position coordinates of the _AUV (x AUV, y _AUV ). , Target point position coordinates (x _goal , y _goal ), current heading angle θ, and target heading angle β, including a total of 11 dimensional inputs.

第２項は、現在のＡＵＶとターゲットポイントの距離に基づいて生じる報酬値ｒ₃₂であり、ＡＵＶが障害物を回避しながらターゲットポイントへ航行するように促し、このため、ＡＵＶがターゲットポイントから離間すると、負の報酬を取得し、ターゲットポイントに使づくと正の報酬を取得し、ＡＵＶがターゲットポイントに到着すると、正の報酬値１．０を取得し、トレーニング周期を終了する。

第３項は、ＡＵＶと現在のターゲットがなす角度αに基づいて生じる報酬ｒ₃₃であり、同様にＡＵＶがターゲットポイントの方向へ航行するように促すが、この項の報酬は、主に、現在のターゲット向首に近くなるように向首角を調整することをＡＵＶに学習させ、経路の長さを減らすようにするためである。
ｒ₃₃＝ｋ_cｃｏｓ（α） （１８）
式中、ｋ_Cは障害物回避過程に対応する報酬係数であり、
最後の総報酬信号はこの３項の報酬値を加重したものに等しく、
ｒ₃＝ｋ₃₁ｒ₃₁＋ｋ₃₂ｒ₃₂＋ｋ₃₃ｒ₃₃ （１９）
式中、ｋ₃₁〜ｋ₃₃はそれぞれ加重値である。 2.3.2) Design of reward function Obstacle avoidance behavior is used to ensure that the AUV avoids suddenly appearing obstacles and arrives at the target point smoothly, so the reward signal component is the third term. _{The first term is the reward value r 31} obtained based on the AUV distance to the obstacle, and as shown in Equation 16, when the AUV approaches the obstacle, a negative reward warning is obtained. When the AUV moves away from the obstacle, it gets a positive reward, encourages the AUV to navigate away from the obstacle, gets a reward value of -1 when it collides with the obstacle, and ends the current training cycle.

_{The second term is the reward value r 32} generated based on the current distance between the AUV and the target point, which encourages the AUV to navigate to the target point while avoiding obstacles, thus causing the AUV to move away from the target point. Then, a negative reward is acquired, a positive reward is acquired when used for the target point, and when the AUV arrives at the target point, a positive reward value of 1.0 is acquired and the training cycle ends.

_{The third term is the reward r 33} generated based on the angle α between the AUV and the current target, which also prompts the AUV to navigate in the direction of the target point, but the reward in this section is mainly the present. This is to train the AUV to adjust the heading angle so that it is closer to the target heading, and to reduce the length of the path.
r ₃₃ = k _c cos (α) (18)
In the formula, k _C is the reward coefficient corresponding to the obstacle avoidance process.
The final total reward signal is equal to the weight of the reward values in these three terms,
r ₃ = k ₃₁ r ₃₁ + k ₃₂ r ₃₂ + k ₃₃ r ₃₃ (19)
In the formula, k _{31 to k 33} are weighted values, respectively.

Further, the process of (3) above is as follows.
Reinforcement learning trains mapping from motion to environment, and it is possible to train mapping from motion to robot model using a robot model as an environment model. Therefore, in the present invention, the kinematics and dynamics model of the robot with fuzzy fluid dynamics parameters, that is, the AUV model is created by directly using the robot as the environment, and the force and torque are obtained through DDPG training to act on the underwater robot, and the AUV model is used. _{The speed and angular velocity of the robot are obtained by calculating using, and the reward value r 4} =-| △ v + △ Ψ | is designed by using the error between the velocity, the angular velocity and the target velocity, and the target angular velocity. , Δv is the velocity error, and ΔΨ is the heading error. It also simulates a dynamically changing underwater environment by adding random interference to the AUV model during training, which gives training a complete DDPG-based control system with anti-interference capabilities. .. After the control system training is completed, the target command is obtained from the robot's current position and target route according to the route tracking strategy, and the DDPG control system is used to control the robot to follow the planned command.

The DDPG control system corresponds to an operating network, and the Deep Policy Policy Gradient (DDPG) is an algorithm that combines Actor Critic and DQN, and improves the stability and convergence of Actor Critic. The concept is to apply the concept of two neural networks in the DQN structure, which have the same structure but different parameter update frequencies, to Actor Critic. Further, using the Deterministic concept, the conventional method of randomly screening the actor critical in the continuous operation section is changed so as to output only two operation values in the continuous space.

式中、Ｑ（ｓ,ａ）は、状態推定ネットワークに基づいて得られるものであり、ａは動作推定ネットワークから伝送してきた動作である。前の部分Ｒ＋γｍａｘ_aＱ（ｓ’,ａ）は現実Ｑ値であり、ＤＱＮと異なるのは、Ｑ値を計算するときに、貪欲法を使用して動作ａ’を選択するのではなく、動作現実ネットワークによりここでのａ’を得るのである。前記のように、Ｃｒｉｔｉｃの状態推定ネットワークのトレーニングは、現実Ｑ値と推定Ｑ値の二乗損失に基づくものであり、推定Ｑ値は、現在の状態ｓに基づいて、動作推定ネットワークにより出力される動作ａの入力状態を推定ネットワークに入力して得るものであり、現実Ｑ値は、現実の報酬Ｒに基づいて、次の時刻の状態ｓ’と、動作現実ネットワークにより得られた動作ａ’を状態現実ネットワークに入力して得たＱ値とを加算して得るものである。 In the Critic system, the learning process of Critic is similar to DQN, and network learning is performed using the loss function of the actual Q value and the estimated Q value as shown in the following equation.

In the equation, Q (s, a) is obtained based on the state estimation network, and a is an operation transmitted from the operation estimation network. The previous part R + γmax _a Q (s', a) is the actual Q value, which is different from DQN in that when calculating the Q value, the action a'is not selected using the greedy algorithm. The a'here is obtained by the real network. As described above, the training of Critic's state estimation network is based on the square loss of the actual Q value and the estimated Q value, and the estimated Q value is output by the motion estimation network based on the current state s. The input state of the operation a is input to the estimation network, and the actual Q value is the state s'at the next time and the operation a'obtained by the operation reality network based on the actual reward R. It is obtained by adding the Q value obtained by inputting to the state-real network.

ｓは状態を表し、ｓ_tはｔ時刻での状態であり、ａは動作を表し、θ^Qとθμはネットワークの重みパラメータを表す。 In the Actor system, the parameters of the motion estimation network are updated based on the following equation.

s represents a state, s _t is the state at t time, a is shows the operation, theta ^Q and θμ represent weighting parameters of the network.

For the same state, if the system outputs two different actions a1 and a2 and the two Q values Q1 and Q2 are fed back from the state estimation network, then if Q1> Q2, using action 1 will result in more. With a reward, in this case, according to the Policy feedback concept, the probability of a1 increases and the probability of a2 decreases, that is, the Actor tries to obtain as large a Q value as possible. Therefore, regarding the loss of the actor, it can be understood that the larger the obtained feedback Q value, the smaller the loss, and the smaller the obtained feedback Q value, the larger the loss. Therefore, the Q value returned from the state estimation network can be used. You can make it negative.

The concept of the DDPG controller is to make the movement in the reinforcement learning algorithm correspond to the thrust and torque of the robot, and the state in the algorithm to correspond to the speed and angular velocity of the robot. By performing learning training on the algorithm, the mapping relationship from force to state is realized.

と組み合わせ、関数で
τ＝μ（ｓ_t｜θμ） （２２）として表し、
ロボット状態ｓは主にロボットの速度と向首として示され、
Ｖ＝［ｕ,ｖ,ｒ］
Ψ＝［０,θ,Ψ］ （２３）
式中、ｕ、ｖ、ｒはそれぞれＡＵＶの縦方向速度、横方向速度及び角速度であり、ΨはＡＵＶの向首角であり、
水平運動であるので、ｖ、ｒは無視され、このため、
τ＝μ（ｓ_t）＝μ（μ（ｔ）,Ψ（ｔ）） （２４）
この式は、制御システムの出力力がロボットの速度、向首及びトリム角がターゲット命令のようになるように制御することを示す。 To apply DDPG the AUV control, first, Critic neural network structure _{Q (s t a t | θ} Q) and Actor neural network structure μ (s _t | θμ), the weighting parameters (theta ^Q and Shitamyu network Show.) Is created. Next, two neural networks, a target network (target_net) and a prediction network (ever_net), are created in the two structures of Critic and Actor, respectively. Next, when the motion output of the DDPG is used as the acting force τ of the control system and the movement of the robot is controlled by the acting force output by the control system, the DDPG control system is mapped from the current state s of the AUV to the force received by the robot. Can be of equation (21)

Combination, the function tau = mu | expressed as _{(s t θμ) (22)} ,
The robot state s is mainly shown as the speed and heading of the robot,
V = [u, v, r]
Ψ = [0, θ, Ψ] (23)
In the equation, u, v, and r are the longitudinal velocity, lateral velocity, and angular velocity of the AUV, respectively, and Ψ is the heading angle of the AUV.
Since it is a horizontal motion, v and r are ignored, and therefore,
τ = μ ( _st ) = μ (μ (t), Ψ (t)) (24)
This equation indicates that the output force of the control system controls the speed, heading and trim angle of the robot to be like a target command.

式中、ｆ−ランダム干渉力；Ｍ−システムの慣性係数行列、Ｍ＝Ｍ_RB+Ｍ_A≧０； Ｍ_RB−キャリアの慣性行列、

且つ

；Ｍ_A−追加品質係数行列、

；

−コリオリ力・求心力係数行列、

；Ｃ_RB−求心力係数行列；

−コリオリ力（モーメント）係数行列、

；

−粘性流体力係数行列、

；τ−制御入力ベクター；ｇ₀−静圧荷重ベクター、研究し易さからゼロとする；

−復元力／トルクベクター。 Embodiment 2
The process of creating an AUV model of fuzzy hydrodynamic parameters according to the first embodiment is a general process of AUV dynamic modeling, which can be realized by using the prior art in the art, and in order to clarify the above process. In the present embodiment, the process of creating an AUV model of fuzzy hydrodynamic parameters will be described, but the present invention includes, but is not limited to, the following method of creating an AUV model of fuzzy hydrodynamic parameters. In the process of creating an AUV model of fuzzy hydrodynamic parameters,
Steps to create a hydrodynamic equation for an underwater robot,

Wherein, f- random interference power; M- system inertia coefficient _{matrix, M = M RB + M A} ≧ 0; M RB - inertia matrix of carrier,

and

; M _A - Add quality coefficient matrix,

;

-Coriolis force / centripetal force coefficient matrix,

C _RB -centripetal force coefficient matrix;

-Coriolis force (moment) coefficient matrix,

;

− Viscous fluid force coefficient matrix,

Τ-Control input vector; g ₀ -Static load vector, zero for ease of study;

-Restoring force / torque vector.

式中、Ｘ、Ｙ、Ｚ、Ｍ、Ｎは、水中ロボットのアクチュエータが発生する作用による水中ロボットの各自由度での力（トルク）を表し、水中ロボットの受ける重力と浮力、スラスターの推力、水中ロボットの動きによる流体動力やいくつかの環境からの外力を含み、
Ｍは水中ロボットの全水中排水量の質量であり、
ｘ_G、ｙ_G、ｚ_Gは水中ロボットの重心の艇体座標系における座標であり、
Ｉ_y、Ｉ_zはそれぞれ艇体座標系のｙ、ｚ軸に対する水中ロボットの質量の慣性モーメントであり、
ｕ、ｖ、ω、ｑ、ｒはそれぞれ水中ロボットの艇体座標系での縦方向速度、横方向速度、垂向速度、トリム角速度、回転角速度であり、

は水中ロボットの艇体座標系での対応する自由度の（角）加速度であり、

などはすべて艇体の一次又は二次流体力学的導関数であり、理論計算、制約付きモデルの試験、識別及び近似推定により得られ得る。 Due to the configuration of the execution mechanism of the autonomous underwater vehicle, its rolling motion is small, and it is thought that ascending / diving, vertical movement, back-and-forth shaking and pitching movement are mainly performed using thrusters, and its kinematic model. Can be approximately described by an equation with five degrees of freedom.

In the formula, X, Y, Z, M, and N represent the force (torque) at each degree of freedom of the underwater robot due to the action generated by the actuator of the underwater robot, and the gravity and buoyancy received by the underwater robot, the thrust of the thruster, and so on. Including fluid power from the movement of underwater robots and external forces from some environments,
M is the mass of the total displacement of the underwater robot,
x _G , y _G , and z _G are the coordinates of the center of gravity of the underwater robot in the hull coordinate system.
I _y and I _z are the moments of inertia of the mass of the underwater robot with respect to the y and z axes of the hull coordinate system, respectively.
u, v, ω, q, and r are the longitudinal velocity, lateral velocity, vertical velocity, trim angular velocity, and rotational angular velocity of the underwater robot in the hull coordinate system, respectively.

Is the (angular) acceleration of the corresponding degree of freedom in the hull coordinate system of the underwater robot,

Etc. are all primary or secondary hydrodynamic derivatives of the hull and can be obtained by theoretical calculations, testing of constrained models, identification and approximate estimation.

Example The main object of the present invention is to free a person from a complicated programming process by autonomously performing action determination and motion control based on the current environmental state in an underwater environment. Specifically, it is realized as follows.

1) Create an autonomous unmanned underwater vehicle action plan simulation system based on deep reinforcement learning using programming software, and obtain the optimal decision strategy for the robot through simulation training.
1.1) Create an environment model, determine the initial position and target point, and initialize the algorithm parameters.
1.2) Determine the environmental state and robot task at the current t time, and decompose the task into target movement behavior, wall tracking behavior, and obstacle avoidance behavior.
1.3) Select movement to target, wall tracking or obstacle avoidance based on the current state to break down the action into actions.
1.4) Execute the operation a, observe the new state s', and obtain the reward value R.
1.5) Train the neural network to obtain the Q value of each motion, and output the motion based on the maximum Q value.
1.6) Update the Q function.
1.7) If the current time state is judged and the target state is reached, 1.8) is entered, and if not, 1.4) is entered.
1.8) Update the Q function after the selected action is completed.
1.9) Judge whether the detection is completed, and if it is completed, enter 1.10), and if not, enter 1.3).
1.10) Judge whether the Q value has converged, and if it has converged, end training or planning, and if it has not converged, initialize the robot position and enter 1.2).

２．２．２．３）ＡＵＶ運動学モデルに基づいてＡＵＶ速度

、向首角

及びトリム角

、速度誤差△ｖ、向首誤差△Ψ及びトリム誤差△θを計算し、報酬戦略に従って報酬値を取得する。
ｒ＝−｜△ｖ＋△Ψ＋△θ｜
２．２．２．４）制御誤差が０であり、ｒ＋＝１である場合、小ループを終了する。
２．２．２．５）ｃｒｉｔｉｃニューラルネットワークを更新して最小損失を取得する。

２．２．２．６）勾配降下法によってａｃｔｏｒを更新する。

２．２．２．７）ネットワークパラメータ

を更新する。
２．２．２．８）内ループのステップ数に達すると、内ループを終了する。
２．２．３）外ループのステップ数に達すると、外ループを終了する。 2) The DDPG controller is used to control the robot to complete the planned and output operation, specifically including the following steps.
2.1) Initialize the parameters.
2.2) Perform an external loop.
2.2.1) Target heading and target speed are randomly generated.
2.2.2) Perform an internal loop.
2.2.2.1) DDPG by algorithm is executed, operation _{τ = a = μ (s t} | θμ) outputs a.
2.2.2.2) Calculate the acceleration of AUV based on the AUV kinematics model.

2.2.2.3) AUV velocity based on AUV kinematics model

, Neck angle

And trim angle

, Speed error Δv, heading error ΔΨ and trim error Δθ are calculated, and the reward value is obtained according to the reward strategy.
r ＝ － ｜ △ v ＋ △ Ψ ＋ △ θ ｜
2.2.2.4) When the control error is 0 and r + = 1, the small loop is terminated.
2.2.2.5) Update the critic neural network to get the minimum loss.

2.2.2.6) Update the actor by the gradient descent method.

2.2.2.7) Network parameters

To update.
2.2.2.8) When the number of steps in the inner loop is reached, the inner loop is terminated.
2.2.3) When the number of steps in the outer loop is reached, the outer loop is terminated.

Claims (6)

AUV action plan and motion control method based on reinforcement learning.
Underwater robot tunnel detection is defined as a total task, or task, and actions to complete the task include moving to a target, wall tracking, and obstacle avoidance, completing the robot-planned action underwater. And the steps to define the concrete control instructions that occur for
When the AUV performs the tunnel detection task, it uses the deep reinforcement learning DQN algorithm to perform action planning in real time based on the underwater environment to be detected, that is, it builds an action planning architecture based on multi-action network calls. Steps to define the input and output behaviors of the environmental state characteristics of the three behaviors according to the needs of the task, build the corresponding deep learning behavioral network, and design the reward function.
The steps that the planning system completes the tunnel detection task by calling the trained behavior network,
Includes steps in which the control system completes planned actions by calling a trained operating network.
The process of building the corresponding deep learning behavioral network and designing the reward function involves the following steps:
The tunnel detection task is broken down into action sequences, multiple feasible route points are planned based on prior environmental information in global route planning, the AUV departs from the placement position and arrives at each route point in sequence.
Since the route point is a global plan under a known environment, during the travel process, the AUV calls the obstacle avoidance action to safely arrive at the route point based on the real-time environmental state, and the AUV is the main tunnel detection task. Call wall tracking action to complete the task according to a given detection target,
The decision module includes global data, decision system, action library and evaluation system, and task information, situation information and planning knowledge are stored in the global data. The decision system is a self-learning planning system combined with the DQN algorithm. , Trained, extract trained network parameters from the behavior library prior to performing the planning task, then plan the current behavior behavior using the current environmental state information as input, and the evaluation system is an enhanced learning algorithm. It is a reward function system of, and each time AUV plans and executes one action action plan, it provides reward based on the state environment and task information, and all the data is stored in the global database.
Among the above actions, the process of moving to the target includes the following steps.
The movement action to the target point is to navigate to the target point while adjusting the head angle when the AUV does not detect an obstacle, and the relationship between the position and angle of the AUV and the target point is mainly used as the feature input amount. In consideration, specifically, the current AUV position coordinates (x _AUV , y _AUV ), the target point coordinates (x _goal , y _goal ), the current heading angle θ, and the target heading angle β are input in a total of 6 dimensions. Among them, the target heading angle β is the heading angle when the AUV is navigating to the target.
Regarding the reward function, when the AUV navigates to the target point in an obstacle-free environment due to the movement action to the target, the reward function is set to the second term.
Item 1 r ₁₁ considers the change in the distance between the AUV and the target point.

The second term r ₁₂ considers the change in the AUV heading angle, and the closer the heading angle is to the target, the larger the reward value of the target value, and the current AUV heading angle α between the target heading angle is α = θ-β (2),
The smaller the absolute value of α, the larger the reward value to be acquired.
r ₁₂ = k _A cos (α) (3)
In the formula, k _A is the reward coefficient corresponding to the process of moving to the target.
The total reward value is the weight of two terms,
r ₁ = k ₁₁ r ₁₁ + k ₁₂ r ₁₂ (4)
In the formula, k ₁₁ and k ₁₂ are weighted values, respectively.
The wall tracking process of the above actions includes the following steps:
The AUV wall tracking behavior takes into account information on the distance and relative angle between the AUV and the wall, and the AUV measures the distance x ₄ and x ₅ of the AUV from the wall through two front and rear ranging sonars located on one side. Acquired,
Obtain the current AUV head angle θ with the compass, estimate the _{current wall angle θ wall, and}

In the equation, l _AUV is the distance between the two sonars in front and behind, and the environmental state feature inputs of the wall tracking behavior are x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , θ, θ _wall and target. The tracking distance is set to _{l goal , where x 1 to x 5} are data measured by three front sonars and front and rear sonars provided on one side, respectively, and the feature input amount is eight, which is AUV. You can fully describe the state relationship between the walls, set a distance threshold to make decisions about sonar data, and if the threshold is exceeded during training, the current training cycle ends.
The reward function is to keep the AUV as parallel to the wall as possible and keep a certain distance from the wall, and the reward signal of the wall tracking action based on the virtual target point is mainly set in the 4th term, and the general wall tracking action. Reward signal is mainly set to 2 items,
The first term considers the angle formed by the AUV and the current wall as in equation (6), and when the angle between the AUV and the wall increases and exceeds the threshold value, a negative reward value is obtained and the angle between the AUV and the wall is obtained. When is reduced, you get a positive reward value,

The second term considers the distance between the front and rear ends of the AUV and the wall as in equation (7), and when the difference between the distance between the AUV and the wall and the preset value decreases, a positive reward is obtained. If the difference increases, you will get a negative reward, the tracking distance can be in the range of ± 0.2m of the preset value, and if the reward value of this item in the tracking range is 0, then with this location The wall distance value is the average of the data from two ranging sonars on the same side.

The total reward r for general wall tracking behavior is the weight of the reward in the second term.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ r ₂₂ (8)
In the formula, k ₂₁ and k ₂₂ are weighted values, respectively.
In tracking based on virtual target points, this virtual target point is a virtual target point created by a wall with an outer right angle and an inner right angle, and if the environment is an outer right angle, when the front sonar is not detecting an obstacle. Since the input to is the maximum detection distance, a virtual wall is built, a virtual target point is added, and if the environment is at right angles, when the forward sonar detects the wall, the AUV faces the other of the current target walls. A virtual target point is built on the side of
The construction of the reward function based on the virtual target points is as follows,

r ₂₄ = k _B cos (α)
In the formula, k _B is the reward factor corresponding to the wall tracking process.
The total reward r for tracking behavior based on virtual target points is the weight of the reward in item 4.
r ₂ = k ₂₁ r ₂₁ + k ₂₂ r ₂₂ + k ₂₃ r ₂₃ + k ₂₄ r ₂₄
In the formula, k ₂₃ and k ₂₄ are weighted values, respectively.
When the AUV is gradually adjusted until it tracks the next part of the wall, for example, if the ranging sonar in the outer right angle environment detects the target wall again, or if the anterior sonar in the inner right angle environment does not detect the further anterior wall. Delete the virtual target point, call the general wall tracking behavior network,
Among the above actions, the obstacle avoidance process includes the following steps.
For the needs of AUV obstacle avoidance behavior, the environmental state feature inputs include data for the three front sonars and front sonars on each side, and the AUV approaches the target point while avoiding obstacles. The feature input further includes a total of 11-dimensional inputs of the current position coordinates (x _AUV , y _AUV ) of the AUV, the target point position coordinates (x _goal , y _goal ), the current heading angle θ, and the target heading angle β. ,
Regarding the reward function, the reward signal is divided into three terms, the first term is the reward value r ₃₁ obtained based on the AUV distance to the obstacle, and when the AUV approaches the obstacle, a negative reward warning is given. Obtain and get a positive reward when the AUV moves away from the obstacle, encourage the AUV to navigate away from the obstacle, get a reward value of -1 when colliding with the obstacle, and set the current training cycle. Finished

_{The second term is the reward value r 32} generated based on the current distance between the AUV and the target point, which encourages the AUV to navigate to the target point while avoiding obstacles, thus causing the AUV to move away from the target point. Then, a negative reward is obtained, a positive reward is obtained when approaching the target point, a positive reward value of 1.0 is obtained when the AUV target point is reached, and the training cycle is completed.

_{The third term is the reward r 33} generated based on the angle α between the AUV and the current target, which also prompts the AUV to navigate in the direction of the target point, but the reward in this section is mainly the present. This is to train the AUV to adjust the heading angle so that it is closer to the target heading, and to reduce the length of the path.
r ₃₃ = k _c cos (α)
In the formula, k _C is the reward coefficient corresponding to the wall obstacle avoidance process.
The final total reward signal is equal to the weight of the reward values in these three terms,
r ₃ = k ₃₁ r ₃₁ + k ₃₂ r ₃₂ + k ₃₃ r ₃₃
In the formula, k _{31 to k 33} are weighted values, respectively.
Reinforcement learning trains the mapping from motion to environment. Using the robot as the environment, the force and torque are obtained through DDPG training to act on the underwater robot, and the speed of the robot is calculated by using the AUV model. _{Obtain the angular velocity and design the reward value r 4} =-| △ v + △ Ψ | using the error between the velocity, the angular velocity and the target velocity, and the target angular velocity, where Δv is the velocity error and ΔΨ is It is a heading error,
In addition, by adding random interference force to the AUV model during training, a control system based on DDPG is obtained by training, and after the training of the control system is completed, the route tracking strategy is performed from the current position and target route of the robot. AUV action planning and motion control method based on enhanced learning, characterized in that a target command is obtained according to the above and the robot is controlled to follow the planning command using a DDPG control system. In the process of building a virtual target points for the outer right angle and the inner perpendicular wall, if the environment is outside a right angle, the position of the virtual target point is determined by the AUV position, ranging sonar data and the safe distance L ₁ The AUV action plan and motion control method based on the reinforcement learning according to claim 1, characterized in that.

In the process of constructing virtual target points for outer and inner right angles walls, if the environment is inner right, the position of the virtual target points is determined by the _{AUV position, head angle and safety distance L 2.} The AUV action plan and motion control method based on the reinforcement learning according to claim 2.

In the process of controlling the robot to obey the planning command using the DDPG control system,
The step in which the DDPG controller maps the movement in the reinforcement learning algorithm to the thrust and torque of the robot, the state in the algorithm to the speed and angular velocity of the robot, performs learning training on the algorithm, and acquires the mapping relationship from force to state. When,
To apply DDPG the AUV control, first, Critic neural network structure _{Q (s t a t | θ} Q) and Actor neural network structure _{μ (s t | θμ) (} θ Q and Shitamyu indicates the weight parameter of the network ) Is created, and two neural networks, the target network target_net and the prediction network equal_net, are created in the two structures of Critic and Actor, respectively, and then the operation output of the DDPG is controlled as the acting force τ of the control system. The movement of the robot is controlled by the acting force output by the system.

In combination with a function, τ = μ | expressed as (s _t θμ),
The robot state s is mainly shown as the speed and heading of the robot,
V = [u, v, r]
Ψ = [0, θ, Ψ]
In the equation, u, v, and r are the longitudinal velocity, lateral velocity, and angular velocity of the AUV, respectively, and Ψ is the heading angle of the AUV.
v, r are ignored,
The equation τ = μ ( _st ) = μ (μ (t), Ψ (t)) controls the output force of the control system so that the speed, heading and trim angle of the robot are like the target command. The AUV action plan and motion control method based on the reinforcement learning according to claim 1, 2 or 3, wherein In the Critic, network learning is performed using the loss function of the actual Q value and the estimated Q value.

In the equation, Q (s, a) is obtained based on the state estimation network, and a is the operation transmitted from the operation estimation network.
R + γmax _a Q (s', a) is the reality Q value, and the reality Q value is the state s'at the next time and the operation a'obtained by the operation reality network based on the actual reward R by the state reality network. The AUV action plan and motion control method based on reinforcement learning according to claim 4, wherein the obtained Q value is added and obtained. The AUV action plan and motion control method based on reinforcement learning according to claim 5, wherein the actor updates the parameters of the motion estimation network based on the following equation.

Images