CN117140527B - Mechanical arm control method and system based on deep reinforcement learning algorithm - Google Patents

Abstract

The present invention discloses a method and system for controlling a robotic arm based on a deep reinforcement learning algorithm, the method comprising: obtaining the perception data of a six-axis robotic arm; based on the reinforcement learning DDPG algorithm, setting an experience replay pool, agent noise exploration and reward function, interactively optimizing and training the perception data to obtain an optimized reinforcement learning DDPG algorithm; deploying the optimized reinforcement learning DDPG algorithm to the six-axis robotic arm for motion guidance. The present invention constructs a reinforcement learning DDPG algorithm by setting an experience replay pool, agent noise exploration and reward function to achieve more accurate target tracking. As a method and system for controlling a robotic arm based on a deep reinforcement learning algorithm, the present invention can be widely used in the field of robotic arm control technology.

Description

A robotic arm control method and system based on deep reinforcement learning algorithm

Technical Field

The present invention relates to the field of robotic arm control technology, and in particular to a robotic arm control method and system based on a deep reinforcement learning algorithm.

Background technique

With the increase of human space activities year by year, the number of non-cooperative space targets such as space debris and failed spacecraft that need to be captured, repaired or cleaned up is increasing day by day. The methods of capturing non-cooperative space targets include rigid solutions and flexible solutions such as rope nets and flying claws. The rigid capture solution mainly involves installing a capture device at the end of the robotic arm, which can accurately capture the target by controlling the robotic arm, making the docking of the service satellite and the target satellite very stable. In addition, the space robotic arm has the advantages of light weight, high flexibility and strong maneuverability, and has broad application prospects in the capture mission of non-cooperative space targets.

Non-cooperative targets have characteristics such as uncertain motion trajectories and easy escape, which are difficult points in the capture process. In the control of manipulators, traditional control methods such as PID control and calculated torque control have the following disadvantages: first, model dependence. Traditional control methods usually rely on accurate system models and parameters. In practical applications, the dynamic model of the manipulator system may be difficult to accurately model. This may cause traditional methods to be unstable or inaccurate in actual operation. Second, insufficient adaptability. Traditional control methods usually have difficulty in dealing with complex manipulator system characteristics such as nonlinearity, time-varying, and coupling. These methods may need to adjust parameters frequently when dealing with complex tasks, and may not be able to share experience between different tasks. Third, parameter adjustment is difficult. Traditional control methods require manual parameter adjustment, which may be a time-consuming and tedious process. When the parameters of the manipulator system change or the tasks change continuously, it may become difficult to maintain and optimize traditional controllers. Fourth, poor generalization ability. Traditional control methods usually have difficulty learning from a small amount of data and have strong generalization ability. When the manipulator system faces unknown environments or tasks, traditional methods may not be able to adapt and learn new strategies.

Summary of the invention

In order to solve the above technical problems, the purpose of the present invention is to provide a robotic arm control method and system based on a deep reinforcement learning algorithm. By setting an experience replay pool, agent noise exploration and reward function, a reinforcement learning DDPG algorithm is constructed to achieve more accurate target tracking.

The first technical solution adopted by the present invention is: a robot arm control method based on a deep reinforcement learning algorithm, comprising the following steps:

Obtain the perception data of the six-axis robotic arm;

Based on the reinforcement learning DDPG algorithm, the experience replay pool, agent noise exploration and reward function are set up to perform interactive optimization training on the perception data to obtain the optimized reinforcement learning DDPG algorithm;

The optimized reinforcement learning DDPG algorithm is deployed to the six-axis robotic arm for motion guidance.

Furthermore, the step of obtaining the perception data of the six-axis robotic arm specifically includes:

Set up the torque input ports and angle and angular velocity sensors for controlling each joint of the six-axis robot arm, and define the state of the intelligent agent as the six joint angles at the end of the six-axis robot arm, joint angular velocity, distance error and velocity error with the target in the x, y, and z directions, and the current control torque of each joint.

Furthermore, the reinforcement learning DDPG algorithm includes a main network and a target network, wherein:

The main network includes an Actor network and a Critic network, and the target network includes a Target Actor network and a Target Critic network;

The Actor network is an action network that takes the state s of the agent as input and outputs the six-degree-of-freedom control torque of the deterministic action robot;

The Critic network is an evaluation network used to calculate the Q value and evaluate the value of the action given by the Actor network through the Q value;

The Target Actor network and the Target Critic network are used to initialize the network parameters of the Actor network and the network parameters of the Critic network, and initialize the network parameters of the Target Actor network and the network parameters of the Target Critic network.

Furthermore, the step of setting up the experience replay pool specifically includes:

The agent stores the obtained experience data (s _t , a _t , r _t , s _t+1 , done) in the experience replay pool, and updates the main network parameters and target network parameters according to batch sampling, where s _t represents the state of the agent at time t, a _t represents the action taken by the agent at time t, r _t represents the reward obtained after taking the action at time t, s _t+1 represents the state reached at time t+1 after the agent takes the action, and done indicates whether the round task has been completed.

Furthermore, the agent noise exploration step specifically includes:

Noise processing is performed on the action output by the Actor network. When extracting samples from the memory bank during network update, the mean square error d between the action a in the current state of the sample and the action a output by the original policy network before the update is calculated and compared with the set error threshold _dth . The standard deviation s of the Gaussian noise is adjusted and updated according to the comparison result.

Furthermore, the expression for adjusting and updating the standard deviation s of the Gaussian noise is:

In the above formula, s represents the standard deviation of Gaussian noise, k represents the adaptive adjustment coefficient, d represents the mean square error value, and _dth represents the set error threshold.

Furthermore, the reward function includes a distance reward function, a speed reward function, a smooth action reward function and an energy loss reward function, wherein:

The distance reward function means that when the distance error between the agent terminal and the target is less than the threshold, the agent obtains a reward positively correlated with the error of being close to the target, otherwise, it obtains a reward negatively correlated with the error. Its expression is:

In the above formula, r ₁ represents the distance reward function, k ₁ represents the adjustable weight coefficient, and e _ri represents the position distance error of the end of the robot arm in the x, y, and z directions respectively;

The speed reward function is used to guide the robot arm to move according to the desired trajectory speed, and its expression is:

In the above formula, r ₂ represents the speed reward function, k ₂ represents the adjustable weight coefficient, and e _vi represents the speed error of the end of the robot in the x, y, and z directions respectively;

The action smoothness reward function represents the control torque penalty, and its expression is:

In the above formula, r ₃ represents the action smoothness reward function, k ₃ represents the adjustable weight coefficient, and torque _i represents the control torque of each axis of the robot arm;

The energy loss reward function represents the torque change penalty, and its expression is:

In the above formula, r ₄ represents the energy loss reward function, k ₄ represents the adjustable weight coefficient, and/> Respectively represent the control torque of each axis of the robot in the current and previous time steps.

Furthermore, the step of interactively optimizing the perception data to obtain an optimized reinforcement learning DDPG algorithm specifically includes:

According to the current state of the six-axis robot arm, the perception data is input into the Actor network to obtain the control torque corresponding to the perception data;

Add noise to the control torque and use it as the actual action of the six-axis robot;

The six-axis robot arm performs the corresponding actual action and updates the state of the six-axis robot arm to obtain the observation state at the next moment;

The current state of the six-axis robot, the reward obtained by executing the corresponding actual action, and the actual action of the six-axis robot are input into the Critic network to obtain the Q estimate;

Input the next moment's observation state into the Target Actor network to obtain the next moment's six-axis robot action output;

The next moment observation state and the next moment six-axis robot action output are input into the Target Critic network to obtain the next moment Q estimation value and reward;

Determine the completion mark of the first interactive optimization of the six-axis robot arm according to the Q estimation value and the Q estimation value at the next moment;

The current state of the six-axis robot, the actual action of the six-axis robot, the reward, the Q estimate at the next moment, and the completion mark of the first interactive optimization are stored in the experience replay pool;

The interactive optimization steps of the above six-axis robot are repeated until the experience replay pool is full, and the optimized reinforcement learning DDPG algorithm is obtained.

The second technical solution adopted by the present invention is: a robotic arm control system based on a deep reinforcement learning algorithm, comprising:

An acquisition module is used to acquire the perception data of the six-axis robot arm;

The interactive module is used to set the experience replay pool, agent noise exploration and reward function based on the reinforcement learning DDPG algorithm, perform interactive optimization training on the perception data, and obtain the optimized reinforcement learning DDPG algorithm;

The deployment module is used to deploy the optimized reinforcement learning DDPG algorithm to the six-axis robot arm for motion guidance.

The beneficial effects of the method and system of the present invention are as follows: the present invention adopts a deep reinforcement learning algorithm, learns and optimizes the control strategy from the perception data, so that the robot arm can make real-time adjustments according to the dynamic information of the target. During the training process, the robot arm interacts with the target and continuously optimizes the strategy through trial and error and reward mechanisms. For dynamic target tracking tasks, on the basis of the DDPG basic framework, adaptive strategy parameters and experience replay improvements are designed, adaptive noise is introduced, and the exploratory nature of the DDPG algorithm is improved to achieve more accurate target tracking.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flowchart of a method for controlling a robotic arm based on a deep reinforcement learning algorithm according to an embodiment of the present invention;

FIG2 is a block diagram of a mechanical arm control system based on a deep reinforcement learning algorithm according to an embodiment of the present invention;

FIG3 is a schematic diagram of a six-axis robotic arm model according to a specific embodiment of the present invention;

FIG4 is a schematic diagram of a linear trajectory of a target space according to a specific embodiment of the present invention;

5 is a schematic diagram of a robot end effector tracking a straight line trajectory according to a specific embodiment of the present invention;

6 is a schematic diagram of coordinate errors of a robot end effector tracking a straight line trajectory according to a specific embodiment of the present invention;

7 is a schematic diagram of a target space circular trajectory according to a specific embodiment of the present invention;

8 is a schematic diagram of a robot end effector tracking a circular trajectory according to a specific embodiment of the present invention;

FIG. 9 is a schematic diagram of coordinate errors of a robot end effector tracking a circular trajectory according to a specific embodiment of the present invention.

Detailed ways

The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

Compared with other traditional control methods, the control method based on deep reinforcement learning does not require an accurate system model and can directly learn control strategies from actual interactions, making it suitable for complex nonlinear and time-varying systems; the capture strategy is highly adaptive and predictive, and the deep reinforcement learning method can adapt to different tasks and environments. By learning in interaction, it can share experience between different tasks and can learn autonomously in actual tasks without manually adjusting parameters, thereby reducing the burden of manual parameter adjustment. It has good applicability to the problem of multi-degree-of-freedom robotic arm capture of non-cooperative targets in space.

Based on the above principles, a deep reinforcement learning algorithm for the dynamic trajectory tracking problem of non-cooperative targets in the space manipulator's non-cooperative target capture task is proposed, and the effectiveness of the algorithm is verified in simulation. For the dynamic target tracking task, based on the DDPG basic framework, adaptive strategy parameters and experience replay improvements are designed to achieve trajectory tracking of the target within the allowable error range. The effectiveness of the reinforcement learning algorithm is verified in the six-degree-of-freedom space manipulator simulation environment, and it is verified that the algorithm has high tracking accuracy in the trajectory tracking task;

The present invention builds a simulation environment for a robotic arm to track a dynamic trajectory target in Matlab's Simscape, and combines Matlab and Simulink for simulation. To facilitate subsequent experiments, the i5 robotic arm of the real six-axis robotic arm AUO is selected, as shown in Figure 3, and the parameters are used as the simulation robotic arm parameters. The joint angle sensor is used to measure the angle and angular velocity of each joint in the process of the robotic arm movement in real time as the state feedback information of deep reinforcement learning. The simulation experiment only considers the end effector tracking to the target position, and the position error and speed error between the end effector and the target point are used as the main indicators for evaluating the control strategy, and does not involve the control of the collision force generated by the contact between the end effector and the target.

1 , the present invention provides a method for controlling a robotic arm based on a deep reinforcement learning algorithm, the method comprising the following steps:

S1, obtain the perception data of the six-axis robot arm;

Specifically, the urdf model of the robot arm is imported into Simulink, and dynamics, kinematics and other properties are added to it. The initial test state of the robot arm is adjusted, and the torque input port and the angle and angular velocity sensors for controlling each joint of the robot arm are set. The hyperparameters of the reinforcement learning agent and the neural network are set in Matlab. The state of the agent is defined as the six joint angles at the end of the robot arm, the joint angular velocity, the distance error and velocity error with the target in the x, y, and z directions, and the current control torque of each joint. Table 1 shows the DH parameters and the range of each joint angle.

Table 1 DH parameters and rotation range of each joint

S2. Based on the reinforcement learning DDPG algorithm, set the experience replay pool, agent noise exploration and reward function, perform interactive optimization training on the perception data, and obtain the optimized reinforcement learning DDPG algorithm;

S21. Set the neural network required by DDPG;

Specifically, the neural network required by DDPG is set, including the main network Actor network, Critic network and target network Target Actor network, Target Critic network with the same network structure. The Actor network is an action network, which takes the state s of the agent as input and outputs the six-degree-of-freedom control torque of the deterministic action robot. The Critic network is an evaluation network, which is used to calculate the Q value. The value of the action given by the Actor network is evaluated by the Q value, which is expressed as Q(s,a,w), where s is the state of the robot, a is the action given by the Actor network, and w is the Critic network parameter. The role of the target network is to slow down the network update speed through soft updates, so that the output will be more stable and ensure that the learning process of the Critic network is smoother. Randomly initialize the network parameters _w1 and _w2 of the Actor network and the Critic network, and initialize the network parameters _w3 and _w4 of the Target Actor network and the Target Critic network. The agent represents the decision function body in the reinforcement learning algorithm, that is, the robot.

S22. Set up experience replay pool;

Specifically, an experience replay pool is set up, and the agent stores the obtained experience data (s _t , a _t , r _t , s _t+1 , done) in the experience replay pool. When updating the network parameters, batch sampling is performed, where s _t represents the state of the agent at time t, a _t represents the action taken by the agent at time t, r _t represents the reward obtained after taking the action at time t, s _t+1 represents the state reached at time t+1 after the agent takes the action, and done represents whether the round task has been completed.

S23, setting agent noise exploration;

Specifically, the agent noise exploration is set, and the action output by the deterministic strategy is a deterministic action, lacking exploration of the environment. In the training stage, noise is added to the action output by the Actor network, so that the agent has a certain exploration ability. Adaptive noise can make the agent explore more in the early stage of training, and gradually reduce exploration in the later stage of training, making better use of the experience already learned. One improvement method is to add Gaussian noise to the action strategy network parameters, that is, to use the strategy parameter noise technique. In the process of extracting samples from the memory bank when the network is updated, the mean square error d between the action a in the current state of the sample and the action a output by the original strategy network before the update is calculated, where the sample represents the data in the experience replay pool, which is compared with the set error threshold _dth , and the standard deviation s of the Gaussian noise is adjusted and updated according to the comparison result. The expression is:

In the above formula, s represents the standard deviation of Gaussian noise, k represents the adaptive adjustment coefficient, d represents the mean square error value, and _dth represents the set error threshold.

S24, constructing a design reward function;

Specifically, the reward function is constructed and designed. In the target tracking operation, the reward function is designed mainly from four aspects: distance, speed, smooth movement and energy loss. It is expected that the final error from the end effector of the manipulator to the target is the smallest, and it is expected that the manipulator's movement amplitude is as small as possible during the movement, and the energy loss is as low as possible.

The distance aspect uses sparse rewards. The specific design is as follows: a distance threshold of 0.01 is set. When the distance error between the end and the target is less than 1/2/5/10 times the threshold, the agent will receive a fixed reward r ₁ = 1000 for successfully approaching the target. times the reward to encourage the agent to gradually approach the goal; otherwise, it will receive a reward negatively correlated with the error:

In the above formula, r ₁ represents the distance reward function, k ₁ represents the adjustable weight coefficient, and e _ri represents the position distance error of the end of the robot arm in the x, y, and z directions respectively;

Speed adoption bonus:

In the above formula, r ₂ represents the speed reward function, k ₂ represents the adjustable weight coefficient, and e _vi represents the speed error of the end of the robot arm in the x, y, and z directions respectively, guiding the robot arm to move as fast as possible according to the desired trajectory speed;

Two penalties are introduced in terms of motion smoothness and energy loss:

Control torque penalty:

In the above formula, r ₃ represents the action smoothness reward function, k ₃ represents the adjustable weight coefficient, and torque _i represents the control torque of each axis of the robot arm, which prevents the control torque input from being too large, causing the robot arm to move a large range, large energy loss, and low training stability;

Torque change penalty:

In the above formula, r ₄ represents the energy loss reward function, k ₄ represents the adjustable weight coefficient, and/> Respectively represent the control torque of each axis of the robot arm at the current and previous time steps, limit the rate of change of the torque given by the agent, and make the movement of the robot arm smoother;

In summary, the reward function is designed as follows:

r＝ _r1 + _r2 + _r3 + _r4

In the above formula, r represents the total reward function.

S25. Interaction optimization.

Specifically, the steps include:

(1) According to the current state of the six-axis robot arm, the perception data is input into the Actor network to obtain the control torque corresponding to the perception data;

(2) Adding noise to the control torque and using it as the actual motion of the six-axis robot;

(3) The six-axis robot arm performs the corresponding actual action and updates the state of the six-axis robot arm to obtain the observation state at the next moment;

(4) Input the current state of the six-axis robot and the actual action of the six-axis robot into the Critic network to obtain the Q estimation value;

(5) Input the observed state at the next moment into the Target Actor network to obtain the action output of the six-axis robot arm at the next moment;

(6) Input the next moment observation state and the next moment six-axis robot arm action output into the TargetCritic network to obtain the next moment Q estimation value and reward;

(7) Determine the completion mark of the first interactive optimization of the six-axis robot arm according to the Q estimation value and the Q estimation value at the next moment;

(8) The current state of the six-axis robot arm, the actual action of the six-axis robot arm, the reward, the Q estimation value at the next moment, and the completion mark of the first interactive optimization are stored in the experience replay pool;

(9) looping the interactive optimization steps (1) to (8) of the six-axis robot arm until the experience replay pool is full, and obtaining the optimized reinforcement learning DDPG algorithm;

That is, through the above settings, the intelligent agent continuously interacts with the environment, repeatedly executing the process of selecting actions, interacting with the environment, storing experience and training the network, so that the intelligent agent gradually learns the optimal strategy.

S3. Deploy the optimized reinforcement learning DDPG algorithm to the six-axis robotic arm for motion guidance.

The present invention takes the reinforcement learning DDPG algorithm as the basic framework, and makes corresponding improvements to the basic DDPG algorithm for the experimental task of tracking dynamic targets to improve the algorithm performance. DDPG is a deep reinforcement learning algorithm used to solve the problem of continuous action space. It combines the Actor-Critic framework and deep neural network, uses Q function as Critic, adopts two algorithm techniques-experience replay (Experience) and target network (Target Net), samples experience data from the experience memory library (Replay Buffer) to update the deep neural network, uses TD error to update the value function network, and outputs actions through deterministic strategies, which can effectively process high-dimensional continuous action space. In order to improve the exploration of the DDPG algorithm, the design introduces adaptive noise. Adaptive noise can make the intelligent agent explore more in the early stage of training, and gradually reduce exploration in the later stage of training, making better use of the experience learned. The reward function plays a vital role in the reinforcement learning algorithm. The present invention constructs and designs a detailed reward function model for the experimental task of tracking dynamic targets, and guides the intelligent agent to learn to adapt and complete the experimental task through the setting of the reward function.

Furthermore, the effectiveness of the deep reinforcement learning algorithm was verified in a six-degree-of-freedom space robot tracking operation simulation environment. The tracking effect of the reinforcement learning algorithm was verified for spatial linear motion targets and spatial curved motion targets. Simulation experiments were conducted on Matlab software. First, the linear trajectory tracking results of the reinforcement learning algorithm were shown. Figure 4 is the linear trajectory of the target space. Figure 5 is the tracking trajectory of the end effector of the robot arm. The robot arm responded quickly to the linear motion target at the initial position. Figure 6 is the coordinate error of the end effector tracking of the robot arm. Next, the circular trajectory tracking results of the reinforcement learning algorithm were shown. Figure 7 is the linear trajectory of the target space. Figure 8 is the tracking trajectory of the end effector of the robot arm. The robot arm responded quickly to the linear motion target at the initial position. Figure 9 is the coordinate error of the end effector tracking of the robot arm.

Referring to FIG2 , a robotic arm control system based on a deep reinforcement learning algorithm includes:

An acquisition module is used to acquire the perception data of the six-axis robot arm;

The interactive module is used to set the experience replay pool, agent noise exploration and reward function based on the reinforcement learning DDPG algorithm, perform interactive optimization training on the perception data, and obtain the optimized reinforcement learning DDPG algorithm;

The deployment module is used to deploy the optimized reinforcement learning DDPG algorithm to the six-axis robot arm for motion guidance.

The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims (2)

1. A robot arm control method based on a deep reinforcement learning algorithm, characterized in that it comprises the following steps: Acquire the perception data of the six-axis robot arm, where the torque input port and the angle and angular velocity sensors for controlling each joint of the six-axis robot arm are set, and the state of the intelligent agent is defined as the six joint angles at the end of the six-axis robot arm, the joint angular velocity, the distance error and velocity error with the target in the x, y, and z directions, and the current control torque of each joint; Based on the reinforcement learning DDPG algorithm, the experience replay pool, agent noise exploration and reward function are set up to perform interactive optimization training on the perception data to obtain the optimized reinforcement learning DDPG algorithm; The reinforcement learning DDPG algorithm includes a main network and a target network, wherein the main network includes an Actor network and a Critic network, and the target network includes a Target Actor network and a Target Critic network; the Actor network is an action network, which takes the state s of the intelligent body as input and outputs the six-degree-of-freedom control torque of the deterministic action manipulator; the Critic network is an evaluation network, which is used to calculate the Q value and evaluate the value of the action given by the Actor network through the Q value; the Target Actor network and the Target Critic network are used to initialize the network parameters of the Actor network and the network parameters of the Critic network, and initialize the network parameters of the Target Actor network and the network parameters of the Target Critic network; The experience replay pool includes: the agent stores the obtained experience data (s _t , a _t , r _t , s _t+1 , done) in the experience replay pool, and updates the main network parameters and the target network parameters according to batch sampling, wherein s _t represents the state of the agent at time t, a _t represents the action taken by the agent at time t, r _t represents the reward obtained after taking the action at time t, s _t+1 represents the state reached at time t+1 after the agent takes the action, and done represents whether the round task has been completed; The agent noise exploration includes adding noise processing to the action output by the Actor network, and in the process of extracting samples from the memory bank when the Actor network is updated, calculating the mean square error d between the action a' in the current state of the sample and the action a output by the Actor network before the update, comparing it with the set error threshold _dth , and adjusting and updating the standard deviation s of the Gaussian noise according to the comparison result, wherein the sample represents the data in the experience replay pool; The expression for adjusting and updating the standard deviation s of Gaussian noise is: In the above formula, s represents the standard deviation of Gaussian noise, k represents the adaptive adjustment coefficient, d represents the mean square error value, and _dth represents the set error threshold; The reward function includes a distance reward function, a speed reward function, a smooth action reward function and an energy loss reward function, wherein: The distance reward function means that when the distance error between the agent terminal and the target is less than the threshold, the agent obtains a reward positively correlated with the error of being close to the target, otherwise, it obtains a reward negatively correlated with the error. Its expression is: In the above formula, r ₁ represents the distance reward function, k ₁ represents the adjustable weight coefficient, and e _ri represents the position distance error of the end of the robot arm in the x, y, and z directions respectively; The speed reward function is used to guide the robot arm to move according to the desired trajectory speed, and its expression is: In the above formula, r ₂ represents the speed reward function, k ₂ represents the adjustable weight coefficient, and e _vi represents the speed error of the end of the robot in the x, y, and z directions respectively; The action smoothness reward function represents the control torque penalty, and its expression is: In the above formula, r ₃ represents the action smoothness reward function, k ₃ represents the adjustable weight coefficient, and torque _i represents the control torque of each axis of the robot arm; The energy loss reward function represents the torque change penalty, and its expression is: In the above formula, r ₄ represents the energy loss reward function, k ₄ represents the adjustable weight coefficient, and/> Respectively represent the control torque of each axis of the robot arm at the current and previous time steps; Among them, interactive optimization training of perception data is performed to obtain the optimized reinforcement learning DDPG algorithm, which also includes: According to the current state of the six-axis robot arm, the perception data is input into the Actor network to obtain the control torque corresponding to the perception data; Add noise to the control torque and use it as the actual action of the six-axis robot; The six-axis robot arm performs the corresponding actual action and updates the state of the six-axis robot arm to obtain the observation state at the next moment; The current state of the six-axis robot, the reward obtained by executing the corresponding actual action, and the actual action of the six-axis robot are input into the Critic network to obtain the Q estimate; Input the observed state at the next moment into the TargetActor network to obtain the six-axis robot arm action output at the next moment; The next moment observation state and the next moment six-axis robot action output are input into the Target Critic network to obtain the next moment Q estimation value and reward; Determine the completion mark of the first interactive optimization of the six-axis robot arm according to the Q estimation value and the Q estimation value at the next moment; The current state of the six-axis robot, the actual action of the six-axis robot, the reward, the Q estimate at the next moment, and the completion mark of the first interactive optimization are stored in the experience replay pool; The interactive optimization steps of the six-axis robot arm are repeated until the experience replay pool is full, and the optimized reinforcement learning DDPG algorithm is obtained; The optimized reinforcement learning DDPG algorithm is deployed to the six-axis robotic arm for motion guidance. 2. A system for a robot arm control method based on a deep reinforcement learning algorithm as claimed in claim 1, characterized in that it comprises the following modules: An acquisition module is used to acquire the perception data of the six-axis robot arm; The interactive module is used to set the experience replay pool, agent noise exploration and reward function based on the reinforcement learning DDPG algorithm, perform interactive optimization training on the perception data, and obtain the optimized reinforcement learning DDPG algorithm; The deployment module is used to deploy the optimized reinforcement learning DDPG algorithm to the six-axis robot arm for motion guidance.