research-article

Air combat maneuver decision based on deep reinforcement learning with auxiliary reward

Authors:

Yongshuai Wang,

Zengqiang ChenAuthors Info & Claims

Neural Computing and Applications, Volume 36, Issue 21

Pages 13341 - 13356

https://doi.org/10.1007/s00521-024-09720-z

Published: 26 April 2024 Publication History

Abstract

For air combat maneuvering decision, the sparse reward during the application of deep reinforcement learning limits the exploration efficiency of the agents. To address this challenge, we propose an auxiliary reward function considering the impact of angle, range, and altitude. Furthermore, we investigate the influences of the network nodes, layers, and the learning rate on decision system, and reasonable parameter ranges are provided, which can serve as a guideline. Finally, four typical air combat scenarios demonstrate good adaptability and effectiveness of the proposed scheme, and the auxiliary reward significantly improves the learning ability of deep Q network (DQN) by leading the agents to explore more intently. Compared with the original deep deterministic policy gradient and soft actor critic algorithm, the proposed method exhibits superior exploration capability with higher reward, indicating that the trained agent can adapt to different air combats with good performance.

References

[1]

Alpdemir MN Tactical UAV path optimization under radar threat using deep reinforcement learning Neural Comput Appl 2022 34 5649-5664

Digital Library

[2]

Liu H, Meng Q, Peng F, and Lewis FL Heterogeneous formation control of multiple UAVs with limited-input leader via reinforcement learning Neurocomputing 2020 412 63-71

[3]

Zhou K, Wei R, and Xu Z An air combat decision learning system based on a brain-like cognitive mechanism Cogn Comput 2020 12 128-139

[4]

Trotta A, Felice MD, Montori F, Chowdhury KR, and Bononi L Joint coverage, connectivity, and charging strategies for distributed UAV networks IEEE Trans Robot 2018 34 883-900

Digital Library

[5]

Sun Z, Wu H, Shi Y, Yu X, Gao Y, Pei W, Yang Z, Piao H, and Hou Y Multi-agent air combat with two-stage graph-attention communication Neural Comput Appl 2023 35 19765-19781

Digital Library

[6]

Shin H, Lee J, Kim H, and Hyunchul Shim D An autonomous aerial combat framework for two-on-two engagements based on basic fighter maneuvers Aerosp Sci Technol 2018 72 305-315

[7]

Maravall Lope J and Fuentes JP Vision-based anticipatory controller for the autonomous navigation of an UAV using artificial neural networks Neurocomputing 2015 151 101-107

[8]

Dai X, Mao Y, and Huang T Automatic obstacle avoidance of quadrotor UAV via CNN-based learning Neurocomputing 2020 402 346-358

[9]

Wang M, Wang L, Yue T, and Liu H Influence of unmanned combat aerial vehicle agility on short-range aerial combat effectiveness Aerosp Sci Technol 2020 96

[10]

Zhou K, Wei R, Xu Z, Zhang Q (2018) (2018) A brain like air combat learning system inspired by human learning mechanism. In: Proceedings of IEEE CSAA guidance, navigation and control conference (CGNCC). IEEE, Xiamen, pp 1–6

[11]

Wang X, Guo K, Chao T, Wang S (2022) Design of differential game guidance law for dual defense aircrafts. In: Proceedings of 2022 5th international symposium on autonomous systems (ISAS). IEEE, Hangzhou, pp 1–6

[12]

Weintraub IE, Pachter M, Garcia E (2020) (2020) An introduction to pursuit-evasion differential games. In: Proceedings of American control conference (ACC). IEEE, Denver, pp 1049–1066

[13]

Ruan W, Sun Y, Deng Y, and Duan H Hawk-pigeon game tactics for unmanned aerial vehicle swarm target defense IEEE Trans Ind Inform 2023 19 11619-11629

[14]

Ma Y, Wang G, Hu X, Luo H, and Lei X Cooperative occupancy decision making of multi-UAV in beyond-visual-range air combat: a game theory approach IEEE Access 2020 8 11624-11634

[15]

Kang Y, Pu Z, Liu Z (2020) (2020) Air-to-air combat tactical decision method based on SIRMs fuzzy logic and improved genetic algorithm. In: Proceedings of international conference on guidance, navigation and control (ICGNC). Springer, Tianjin, pp 3699–3709

[16]

Crumpacker JB, Robbins MJ, and Jenkins PR An approximate dynamic programming approach for solving an air combat maneuvering problem Expert Syst Appl 2022 203

Digital Library

[17]

Sharma R (2014) (2014) Fuzzy Q learning based UAV autopilot. In: Proceedings of innovative applications of computational intelligence on power, energy and controls with their impact on humanity (CIPECH). IEEE, Ghaziabad, pp 29–33

[18]

Liu Y, Liu W, Obaid MA, and Abbas IA Exponential stability of Markovian jumping Cohen–Grossberg neural networks with mixed mode-dependent time-delays Neurocomputing 2016 177 409-415

Digital Library

[19]

Du B, Liu Y, and Atiatallah Abbas I Existence and asymptotic behavior results of periodic solution for discrete-time neutral-type neural networks J Frankl Inst 2016 353 448-461

[20]

Emuna R, Duffney R, Borowsky A, and Biess A Example-guided learning of stochastic human driving policies using deep reinforcement learning Neural Comput Appl 2022 35 16791-16804

Digital Library

[21]

Kiani F and Saraç ÖF A novel intelligent traffic recovery model for emergency vehicles based on context-aware reinforcement learning Inf Sci 2023 619 288-309

Digital Library

[22]

Damadam S, Zourbakhsh M, Javidan R, and Faroughi A An intelligent IoT based traffic light management system: deep reinforcement learning Smart Cities 2022 5 1293-1311

[23]

Zhu R, Li L, Wu S, Lv P, Li Y, and Xu M Multi-agent broad reinforcement learning for intelligent traffic light control Inf Sci 2023 619 509-525

Digital Library

[24]

Du G, Zou Y, Zhang X, Liu T, Wu J, and He D Deep reinforcement learning based energy management for a hybrid electric vehicle Energy 2020 201

[25]

Yang D, Karimi HR, and Pawelczyk M A new intelligent fault diagnosis framework for rotating machinery based on deep transfer reinforcement learning Control Eng Pract 2023 134

[26]

Liu Q, Shi L, Sun L, Li J, Ding M, and Shu FS Path planning for UAV-mounted mobile edge computing with deep reinforcement learning IEEE Trans Veh Technol 2020 69 5723-5728

[27]

Hoel C-J, Driggs-Campbell K, Wolff K, Laine L, and Kochenderfer MJ Combining planning and deep reinforcement learning in tactical decision making for autonomous driving IEEE Trans Intell Veh 2020 5 294-305

[28]

Leong AS, Ramaswamy A, Quevedo DE, and Karl H Deep reinforcement learning for wireless sensor scheduling in cyber-physical system Automatic 2020 113 108759

Digital Library

[29]

Liessner R, Schmitt J, Dietermann A, Bäker B (2019) Hyperparameter optimization for deep reinforcement learning in vehicle energy management. In: Proceedings of 11th international conference on agents artificial intelligence SCITEPRESS—science and technology publications, Prague, pp 134–144

[30]

Chen Y, Zhang J, Yang Q, Zhou Y, Shi G, Wu Y (2020) Design and verification of UAV maneuver decision Simulation system based on deep Q-learning network. In: Proceedings of 2020 16th international conference on control, automation, robotics and vision (ICARCV). IEEE, Shenzhen, pp 817–823

[31]

Cao Y, Kou Y-X, Li Z-W, and Xu A Autonomous maneuver decision of UCAV air combat based on double deep Q network algorithm and stochastic game theory Int J Aerosp Eng 2023 2023 1-20

[32]

Zhang J, Yu Y, Zheng L, Yang Q, Shi G, and Wu Y Situational continuity-based air combat autonomous maneuvering decision-making Def Technol 2023 29 66-79

[33]

Yang Q, Zhu Y, Zhang J, Qiao S, Liu J (2019) UAV air combat autonomous maneuver decision based on DDPG algorithm. In: 2019 IEEE 15th international conference on control automation. ICCA. IEEE, Edinburgh, pp 37–42

[34]

Zhang J, Yang Q, and Shi G UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning J Syst Eng Electron 2021 32 1421-1438

[35]

Wang Z, Guo Y, Li N, Hu S, and Wang M Autonomous collaborative combat strategy of unmanned system group in continuous dynamic environment based on PD-MADDPG Comput Commun 2023 200 182-204

Digital Library

[36]

Li L, Zhang X, Qian C, et al. Basic flight maneuver generation of fixed-wing plane based on proximal policy optimization Neural Comput Appl 2023 2023 1-17

[37]

Wang Z, Li H, Wu Z, and Wu H A pretrained proximal policy optimization algorithm with reward shaping for aircraft guidance to a moving destination in three-dimensional continuous space Int J Adv Robot Syst 2021 18 172988142198954

[38]

Liu X, Yin Y, Su Y, and Ming R A multi-UCAV cooperative decision-making method based on an MAPPO algorithm for beyond-visual-range air combat Aerospace 2022 9 563-582

[39]

Xu J, Zhang J, Yang L, and Liu C Autonomous decision-making for dogfights based on a tactical pursuit point approach Aerosp Sci Technol 2022 129

[40]

Li B, Bai S, Liang S, Ma R, Neretin E, and Huang J Manoeuvre decision-making of unmanned aerial vehicles in air combat based on an expert actor-based soft actor critic algorithm CAAI Trans Intell Technol 2023 8 1608-1619

Digital Library

[41]

Li B, Huang J, Bai S, Gan Z, Liang S, Evgeny N, and Yao S Autonomous air combat decision-making of UAV based on parallel self-play reinforcement learning CAAI Trans Intell Technol 2023 8 64-81

Digital Library

[42]

Huang C, Dong K, Huang H, and Tang S Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization J Syst Eng Electron 2018 29 86-97

[43]

Johnson J Automating the OODA loop in the age of intelligent machines: reaffirming the role of humans in command-and-control decision-making in the digital age Def Stud 2023 23 43-67

[44]

Wang LX, Guo YG, Zhang Q, and Yue T Suggestion for aircraft flying qualities requirements of a short-range air combat mission Chin J Aeronaut 2017 30 881-897

[45]

Li Y, Lyu Y, Shi J, and Li W Autonomous maneuver decision of air combat based on simulated operation command and FRV-DDPG algorithm Aerospace 2022 9 658-676

[46]

Austin F, Carbone G, Falco M, Hinz H, Lewis M (1987) Automated maneuvering decisions for air-to-air combat. In: Guidance, navigation and control conference, pp 2393

Cited By

Zhang YYou SYan YOu QGao XJiang F(2024)A transfer learning model for cognitive electronic reconnaissance of unmanned aerial vehicleEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109158137:PAOnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.109158

Recommendations

Decision Modeling and Simulation of Fighter Air-to-ground Combat Based on Reinforcement Learning
IPMV '22: Proceedings of the 4th International Conference on Image Processing and Machine Vision

With the Artificial Intelligence (AI) widely used in air combat simulation system, the decision-making system of fighter has reached a high level of complexity. Traditionally, the pure theoretical analysis and the rule-based system are not enough to ...
Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction
Abstract
Nowadays, various innovative air combat paradigms that rely on unmanned aerial vehicles (UAVs), i.e., UAV swarm and UAV-manned aircraft cooperation, have received great attention worldwide. During the operation, UAVs are expected to perform agile ...
Discovering Expert-Level Air Combat Knowledge via Deep Excitatory-Inhibitory Factorized Reinforcement Learning
Artificial Intelligence (AI) has achieved a wide range of successes in autonomous air combat decision-making recently. Previous research demonstrated that AI-enabled air combat approaches could even acquire beyond human-level capabilities. However, there ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neural Computing and Applications

Neural Computing and Applications Volume 36, Issue 21

Jul 2024

733 pages

EISSN:1433-3058

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 26 April 2024

Accepted: 25 March 2024

Received: 12 August 2023

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YYou SYan YOu QGao XJiang F(2024)A transfer learning model for cognitive electronic reconnaissance of unmanned aerial vehicleEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109158137:PAOnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.109158

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents