Cited By
View all- Li TLong QChai HZhang SJiang FLiu HHuang WJin DLi Y(2025)Generative AI Empowered Network Digital Twins: Architecture, Technologies, and ApplicationsACM Computing Surveys10.1145/371168257:6(1-43)Online publication date: 10-Jan-2025
Spectrum-Energy-Efficient Mode Selection and Resource Allocation for Heterogeneous V2X Networks: A Federated Multi-Agent Deep Reinforcement Learning Approach
Pages 2689 - 2704
Abstract
Heterogeneous communication environments and broadcast feature of safety-critical messages bring great challenges to mode selection and resource allocation problem. In this paper, we propose a federated multi-agent deep reinforcement learning (DRL) scheme with action awareness to solve mode selection and resource allocation problem for ensuring quality of service (QoS) in heterogeneous V2X environments. The proposed scheme includes an action-observation-based DRL and a model parameter aggregation algorithm considering local model historical parameters. By observing the actions of adjacent agents and dynamically balancing the historical samples of rewards, the action-observation-based DRL can ensure fast convergence of each agent’ individual model. By randomly sampling historical model parameters and adding them to the foundation model aggregation process, the model parameter aggregation algorithm improves foundation model generalization. The generalized model is only sent to each new agent, so each old agent can retain the personality of its individual model. Simulation results show that the proposed scheme outperforms the comparison algorithms in the key performance indicators.
References
[1]
Z. Pei, W. Chen, C. Li, L. Du, H. Liu, and X. Wang, “Analysis and optimization of multihop broadcast communication in the Internet of Vehicles based on C-V2X mode 4,” IEEE Sensors J., vol. 22, no. 12, pp. 12428–12443, Jun. 2022.
[2]
J. B. Kenney, “Dedicated short-range communications (DSRC) standards in the United States,” Proc. IEEE, vol. 99, no. 7, pp. 1162–1182, Jul. 2011.
[3]
S. Gyawali, S. Xu, Y. Qian, and R. Q. Hu, “Challenges and solutions for cellular based V2X communications,” IEEE Commun. Surveys Tuts., vol. 23, no. 1, pp. 222–255, 1st Quart., 2020.
[4]
A. Bazzi, A. O. Berthet, C. Campolo, B. M. Masini, A. Molinaro, and A. Zanella, “On the design of sidelink for cellular V2X: A literature review and outlook for future,” IEEE Access, vol. 9, pp. 97953–97980, 2021.
[5]
X. Zhang, M. Peng, S. Yan, and Y. Sun, “Deep-reinforcement-learning-based mode selection and resource allocation for cellular V2X communications,” IEEE Internet Things J., vol. 7, no. 7, pp. 6380–6391, Jul. 2020.
[6]
H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep reinforcement learning based resource allocation for V2V communications,” IEEE Trans. Veh. Technol., vol. 68, no. 4, pp. 3163–3173, Apr. 2019.
[7]
D. Zhao, H. Qin, B. Song, Y. Zhang, X. Du, and M. Guizani, “A reinforcement learning method for joint mode selection and power adaptation in the V2V communication network in 5G,” IEEE Trans. Cogn. Commun. Netw., vol. 6, no. 2, pp. 452–463, Jun. 2020.
[8]
L. Liang, H. Ye, and G. Y. Li, “Spectrum sharing in vehicular networks based on multi-agent reinforcement learning,” IEEE J. Sel. Areas Commun., vol. 37, no. 10, pp. 2282–2292, Oct. 2019.
[9]
A. D. Mafuta, B. T. J. Maharaj, and A. S. Alfa, “Decentralized resource allocation-based multiagent deep learning in vehicular network,” IEEE Syst. J., vol. 17, no. 1, pp. 87–98, Mar. 2023.
[10]
Y.-H. Xu, C.-C. Yang, M. Hua, and W. Zhou, “Deep deterministic policy gradient (DDPG)-based resource allocation scheme for NOMA vehicular communications,” IEEE Access, vol. 8, pp. 18797–18807, 2020.
[11]
O. A. Wahab, A. Mourad, H. Otrok, and T. Taleb, “Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems,” IEEE Commun. Surveys Tuts., vol. 23, no. 2, pp. 1342–1397, 2nd Quart., 2021.
[12]
W. Sun, D. Yuan, E. G. Ström, and F. Brännström, “Cluster-based radio resource management for D2D-supported safety-critical V2X communications,” IEEE Trans. Wireless Commun., vol. 15, no. 4, pp. 2756–2769, Apr. 2016.
[13]
L. Liang, S. Xie, G. Y. Li, Z. Ding, and X. Yu, “Graph-based resource sharing in vehicular communication,” IEEE Trans. Wireless Commun., vol. 17, no. 7, pp. 4579–4592, Jul. 2018.
[14]
M. Peng, Y. Li, T. Q. S. Quek, and C. Wang, “Device-to-device underlaid cellular networks under Rician fading channels,” IEEE Trans. Wireless Commun., vol. 13, no. 8, pp. 4247–4259, Aug. 2014.
[15]
L. Liang, J. Kim, S. C. Jha, K. Sivanesan, and G. Y. Li, “Spectrum and power allocation for vehicular communications with delayed CSI feedback,” IEEE Wireless Commun. Lett., vol. 6, no. 4, pp. 458–461, Aug. 2017.
[16]
X. Li, L. Ma, R. Shankaran, Y. Xu, and M. A. Orgun, “Joint power control and resource allocation mode selection for safety-related V2X communication,” IEEE Trans. Veh. Technol., vol. 68, no. 8, pp. 7970–7986, Aug. 2019.
[17]
C. Wu, T. Yoshinaga, Y. Ji, and Y. Zhang, “Computational intelligence inspired data delivery for vehicle-to-roadside communications,” IEEE Trans. Veh. Technol., vol. 67, no. 12, pp. 12038–12048, Dec. 2018.
[18]
S. Yan, X. Zhang, H. Xiang, and W. Wu, “Joint access mode selection and spectrum allocation for fog computing based vehicular networks,” IEEE Access, vol. 7, pp. 17725–17735, 2019.
[19]
R. F. Atallah, C. M. Assi, and M. J. Khabbaz, “Scheduling the operation of a connected vehicular network using deep reinforcement learning,” IEEE Trans. Intell. Transp. Syst., vol. 20, no. 5, pp. 1669–1682, May 2019.
[20]
T. Dang and M. Peng, “Joint radio communication, caching, and computing design for mobile virtual reality delivery in fog radio access networks,” IEEE J. Sel. Areas Commun., vol. 37, no. 7, pp. 1594–1607, Jul. 2019.
[21]
K. Zhang, Y. Zhu, S. Leng, Y. He, S. Maharjan, and Y. Zhang, “Deep learning empowered task offloading for mobile edge computing in urban informatics,” IEEE Internet Things J., vol. 6, no. 5, pp. 7635–7647, Oct. 2019.
[22]
H. Ren and M. Q.-H. Meng, “Game-theoretic modeling of joint topology control and power scheduling for wireless heterogeneous sensor networks,” IEEE Trans. Autom. Sci. Eng., vol. 6, no. 4, pp. 610–625, Oct. 2009.
[23]
K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” in Handbook of Reinforcement Learning and Control. Cham, Switzerland: Springer, 2021, pp. 321–384.
[24]
K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Basar, “Fully decentralized multi-agent reinforcement learning with networked agents,” in Proc. Int. Conf. Mach. Learn., 2018, pp. 5872–5881.
[25]
Q. Tang, R. Xie, F. R. Yu, T. Huang, and Y. Liu, “Decentralized computation offloading in IoT fog computing system with energy harvesting: A Dec-POMDP approach,” IEEE Internet Things J., vol. 7, no. 6, pp. 4898–4911, Jun. 2020.
[26]
T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” J. Mach. Learn. Res., vol. 21, no. 1, pp. 7234–7284, 2020.
[27]
Z. Guo, Z. Chen, P. Liu, J. Luo, X. Yang, and X. Sun, “Multi-agent reinforcement learning-based distributed channel access for next generation wireless networks,” IEEE J. Sel. Areas Commun., vol. 40, no. 5, pp. 1587–1599, May 2022.
[28]
X. Lu, L. Xiao, T. Xu, Y. Zhao, Y. Tang, and W. Zhuang, “Reinforcement learning based PHY authentication for VANETs,” IEEE Trans. Veh. Technol., vol. 69, no. 3, pp. 3068–3079, Mar. 2020.
[29]
X. Wang, Y. Han, C. Wang, Q. Zhao, X. Chen, and M. Chen, “In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning,” IEEE Netw., vol. 33, no. 5, pp. 156–165, Sep. 2019.
[30]
R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in Proc. Adv. Neural Inf. Process. Syst., vol. 30, 2017, pp. 6379–6390.
[31]
P. Sunehaget al., “Value-decomposition networks for cooperative multi-agent learning,” in Proc. Int. Conf. Auton. Agents MultiAgent Syst. (AAMAS), 2018, pp. 2085–2087.
Recommendations
Deep reinforcement learning for multi-agent interaction
Multi-agent systems research in the United KingdomThe development of autonomous agents which can interact with other agents to accomplish a given task is a core area of research in artificial intelligence and machine learning. Towards this goal, the Autonomous Agents Research Group develops novel ...
Multi-agent deep reinforcement learning: a survey
AbstractThe advances in reinforcement learning have recorded sublime success in various domains. Although the multi-agent domain has been overshadowed by its single-agent counterpart during this progress, multi-agent reinforcement learning gains rapid ...
Energy-efficient joint resource allocation in 5G HetNet using Multi-Agent Parameterized Deep Reinforcement learning
AbstractSmall cells are a promising technique to improve the capacity and throughput of future wireless networks. However, user association and power allocation in heterogeneous networks is complicated by the dense deployment of small cells, resulting in ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1063-6692 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 13 February 2024
Published in TON Volume 32, Issue 3
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 16Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)4
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Li TLong QChai HZhang SJiang FLiu HHuang WJin DLi Y(2025)Generative AI Empowered Network Digital Twins: Architecture, Technologies, and ApplicationsACM Computing Surveys10.1145/371168257:6(1-43)Online publication date: 10-Jan-2025
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in