Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

UAV-enabled fair offloading for MEC networks: a DRL approach based on actor-critic parallel architecture

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Data processing is a key challenge for computationally limited Ground Users (GUs) in various applications. Unmanned Aerial Vehicles (UAVs) equipped with Multi-access Edge Computing (MEC) servers can assist GUs by offloading their computing tasks. However, existing work ignores fairness when multiple GUs compete for limited computing resources, which may result in UAV underserving certain GUs. In this paper, we investigate a flight trajectory optimization based on reinforcement learning for UAV selection of target GUs for task computation, which provides low latency and fair offloading computing services for GUs by jointly training UAV flight trajectories and task offloading decisions. We formulate UAV flight and offloading as a mixed integer non-convex optimization problem with high-dimensional state and action spaces. The problem is then transformed into a Markov Decision Processes (MDPs) problem and the Maximizing Service Efficiency Proximal Policy Optimization (MSE-PPO) algorithm is proposed to find the optimal solution. The algorithm adopts an actor-critic-based parallel architecture to handle the parameterized action space. Specifically, the UAV position sequence is updated while ensuring an optimal offloading policy between the UAV and the GUs. Simulation results verify that the average system rewards including computational energy efficiency and fairness index are improved by 35.06\(\%\) and 12.10\(\%\) respectively compared to DDPG and PPO algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets generated during and/or analysed during the current study are not publicly available due to [REASON(S) WHY DATA ARE NOT PUBLIC] but are available from the corresponding author on reasonable request.

References

  1. Cui J, Wei L, Zhong H, Zhang J, Xu Y, Liu L (2020) Edge computing in VANETs-An efficient and privacy-preserving cooperative downloading scheme. IEEE J Sel Areas Commun 38:1191–1204

  2. Sharma MK, Kumar M, Saini J (2021) Design and analysis of a compact UWB-MIMO antenna with improved isolation for UWB/WLAN applications. Wirel Pers Commun 119:2913–2928

  3. Senthilkumar G, Tamilarasi K, Kaviarasan S, Arun M (2022) Trusty authentication of devices using blockchain-cloud of things (B-CoT) for fulfilling commercial services. Int J Syst Assur Eng 1–11

  4. Anwar MR, Wang S, Akram MF, Raza S, Mahmood S (2022) 5G-Enabled MEC: a distributed traffic steering for seamless service migration of internet of vehicles. IEEE Internet Things J 9:648–661

  5. Ju Y, Chen Y, Cao Z, Liu L, Pei Q, Xiao M, Ota K, Dong M, Leung V (2023) Joint secure offloading and resource allocation for vehicular edge computing network: a multi-agent deep reinforcement learning approach. IEEE Trans Intell Transp Syst , in press

  6. Huang S, Zhang J, Wu Y (2022) Altitude optimization and task allocation of UAV-Assisted MEC communication system. Sensors 22:8061

  7. Liu Y, Xiong K, Ni Q, Fan P, Letaief KB (2020) UAV-Assisted wireless powered cooperative mobile edge computing: joint offloading, CPU control, and trajectory optimization. IEEE Internet Things J 7:2777–2790

  8. Li H, Wu S, Jiao J, Lin XH, Zhang N, Zhang Q (2023) Energy-efficient task offloading of edge-aided maritime UAV systems. IEEE Trans Veh Technol 72:1116–1126

  9. Zhang K, Gui X, Ren D, Li D (2021) Energy-latency tradeoff for computation offloading in uav-assisted multiaccess edge computing system. IEEE Internet Things J 8:6709–6719

  10. Zhang P, Wang C, Jiang C, Benslimane A (2021) UAV-Assisted multi-access edge computing: technologies and challenges. IEEE Internet Things Mag 4:12–17

  11. Akbari M, Syed A, Kennedy W, Erol-Kantarci M (2023) Constrained federated learning for AoI-limited SFC in UAV-Aided MEC for smart agriculture. IEEE Trans Mach Learn Commun Network 1:277–295

  12. Zhang T, Xu Y, Loo J, Yang D, Xiao L (2020) Joint computation and communication design for UAV-assisted mobile edge computing in IoT. IEEE Trans Ind Inform 16:5505–5516

  13. Li Q, Shi L, Zhang Z, Zheng G (2023) Resource Allocation in UAV-enabled wireless-powered MEC networks with hybrid passive and active communications. IEEE Internet Things J 10:2574–2588

  14. Ye W, Luo J, Shan F, Wu W, Yang M (2020) Offspeeding: optimal energy-efficient flight speed scheduling for UAV-assisted edge computing. Comput. Networks 183:107577

  15. Zhang T, Xu Y, Loo JK, Yang D, Xiao L (2019) Joint computation and communication design for UAV-assisted mobile edge computing in IoT. IEEE Trans Industr Inform 16:5505–5516

  16. Sun H, Zhou F, Hu RQ (2019) Joint offloading and computation energy efficiency maximization in a mobile edge computing system. IEEE Trans Veh Technol 68:3052–3056

  17. Lin X, Bi S, Cheng N, Dai M, Wang H (2022) An \(\alpha \)-fairness approach to balancing the energy consumption among sensors for UAV-IoT systems. IEEE Internet Things J 9:17965–17978

  18. Ding Z, Xu D, Schober R, Poor HV (2022) Hybrid NOMA offloading in multi-user MEC networks. IEEE Trans Wirel Commun 21:5377–5391

  19. Chi HR, Radwan A (2023) Fully-decentralized fairness-aware federated MEC small-cell peer-offloading for enterprise management networks. IEEE Trans Industr Inform 19:644–652

  20. Ren D, Gui X, Zhang K (2022) Adaptive request scheduling and service caching for MEC-assisted IoT networks: an online learning approach. IEEE Internet Things J 9:17372–17386

  21. Sun Y, Xu J, Cui S (2022) User association and resource allocation for MEC-enabled IoT networks. IEEE Trans Wirel Commun 21:8051–8062

  22. Liu M, Feng G, Sun Y, Chen N, Tan W (2023) A network function parallelism-enabled MEC framework for supporting low-latency services. IEEE Trans Serv Comput 16:40–52

  23. Lekharu A, Jain M, Sur A, Sarkar A (2022) Deep learning model for content aware caching at MEC servers. IEEE Trans Netw Service Manag 19:1413–1425

  24. Hua M, Yang L, Pan C, Nallanathan A (2019) Throughput maximization for full-duplex UAV aided small cell wireless systems. IEEE Wireless Commun Lett 9:475–479

  25. Wu Q, Zeng Y, Zhang R (2017) Joint trajectory and communication design for multi-UAV enabled wireless networks. IEEE Trans Wirel Commun 17:2109–2121

  26. Zhan C, Zeng Y, Zhang R (2018) Energy-efficient data collection in UAV enabled wireless sensor network. IEEE Wireless Commun Lett 7:328–331

  27. Wu Q, Zhang R (2018) Common throughput maximization in UAV-enabled OFDMA systems with delay consideration. IEEE Trans Commun 66:6614–6627

  28. Wang Q, Gao A, Hu Y (2021) Joint power and QoE optimization scheme for multi-UAV assisted offloading in mobile computing. IEEE Access 9:21206–21217

  29. Zhong X, Huo Y, Dong X, Liang Z (2020) QoS-compliant 3-D deployment optimization strategy for UAV base stations. IEEE Syst J 15:1795–1803

  30. Xu Y, Zhang T, Liu Y et al (2022) Cellular-connected multi-UAV MEC networks: an online stochastic optimization approach. IEEE Trans Commun 70:6630–6647

  31. Gu B, Zhang X, Lin Z, Alazab M (2021) Deep multi-agent reinforcement-learning-based resource allocation for internet of controllable things. IEEE Internet Things J 8:3066–3074

  32. Li J, Yi C, Chen J, Zhu K, Cai J (2023) Joint trajectory planning, application placement, and energy renewal for UAV-assisted MEC: A triple-learner-based approach. IEEE Internet Things J 10:13622–13636

  33. Song F et al (2023) Evolutionary multi-objective reinforcement learning based trajectory control and task offloading in UAV-assisted mobile edge computing. IEEE Trans Mobile Comput 22:7387–7405

  34. Nie Y, Zhao J, Gao F, Yu FR (2021) Semi-distributed resource management in UAV-aided MEC systems: a multi-agent federated reinforcement learning approach. IEEE Trans Veh Technol 70:13162–13173

  35. Wang Y, Fang W, Ding Y, Xiong NN (2021) Computation offloading optimization for UAV-assisted mobile edge computing: a deep deterministic policy gradient approach. Wirel Networks 27:2991–3006

  36. Zhou X, Huang L, Ye T, Sun W (2022) Computation bits maximization in UAV-assisted MEC networks with fairness constraint. IEEE Internet Things J 9:20997–21009

  37. Mnih V, Kavukcuoglu K, Silver D, Rusu A, Veness J, Bellemare MG et al (2015) Human-level control through deep reinforcement learning. Nat 518:529–533

  38. Tuli S, Ilager S, Ramamohanarao K, Buyya R (2022) Dynamic scheduling for stochastic edge-cloud computing environments using A3C learning and residual recurrent neural networks. IEEE Trans Mobile Comput 21:940–954

  39. Lu S, Liu S, Zhu Y, Liang W, Li K, Lu Y (2023) A DRL-based decentralized computation offloading method: an example of an intelligent manufacturing scenario. IEEE Trans Industr Inform 19:9631–9641

  40. Chen Z, Yin B, Zhu H, Li Y, Tao M, Zhang W (2022) Mobile communications, computing, and caching resources allocation for diverse services via multi-objective proximal policy optimization. IEEE Trans Commun 70:4498–4512

  41. Ho Tai, Nguyen K, Cheriet M (2021) UAV control for wireless service provisioning in critical demand areas: a deep reinforcement learning approach. IEEE Trans Veh Technol 70:7138–7152

  42. Mao K, Zhu Q, Qiu Y, Liu X, Song M, Fan W, Kokkeler A, Miao Y (2023) A UAV-aided real-time channel sounder for highly dynamic nonstationary A2G scenarios. IEEE Trans Instrum Meas 72:1–15

  43. Liu C, Dai Z, Zhao Y, Crowcroft J, Wu D, Leung K (2021) Distributed and energy-efficient mobile crowdsensing with charging stations by deep reinforcement learning. IEEE Trans. Mobile Comput 20:130–146

  44. Zhao M, Li W, Bao L, Luo J, He Z, Liu D (2021) Fairness-aware task scheduling and resource allocation in UAV-enabled mobile edge computing networks. IEEE Trans Green Commun Netw 5:2174–2187

  45. Wang Y, Chen M, Pan C, Wang K, Pan Y (2022) Joint optimization of UAV trajectory and sensor uploading powers for UAV-assisted data collection in wireless sensor networks. IEEE Internet Things J 9:11214–11226

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (62176088, 62303159), International Strategic Innovative Project of National Key Research & Development Program of China (2023YFE0112500), Key Project of Science and Technology Research of the Education Department of Henan Province (22A120001), China Postdoctoral Science Foundation Funded Project (2023M741008).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Henan University, Zhengzhou, 450046, People’s Republic of China

    Wei Li, Si Li, Huaguang Shi, Wenhao Yan & Yi Zhou

  2. International Joint Research Laboratory for Cooperative Vehicular Networks of Henan, Zhengzhou, 450046, People’s Republic of China

    Wei Li, Si Li, Huaguang Shi, Wenhao Yan & Yi Zhou

Authors
  1. Wei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Si Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huaguang Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenhao Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Wei Li, Si Li, Huaguang Shi, Wenhao Yan and Yi Zhou. The first draft of the manuscript was written by Huaguang Shi and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Huaguang Shi.

Ethics declarations

Conflict of Interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled, “UAV-enabled Fair Offloading for MEC Networks: A DRL Approach based on Actor-Critic Parallel Architecture”.

Ethical and informed consent for data used

The data used did not involve human participants and animal studies.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, W., Li, S., Shi, H. et al. UAV-enabled fair offloading for MEC networks: a DRL approach based on actor-critic parallel architecture. Appl Intell 54, 3529–3546 (2024). https://doi.org/10.1007/s10489-024-05339-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05339-8

Keywords

Search

Navigation