Cooperative Spectrum Handoff based on Counterfactual Multi-agent Method
Authors: 
Shufeng Li, Wei Shao, Yunfei PengAuthors Info & Claims
Pages 84 - 90
Abstract
Cognitive radio networks (CRNs) improve spectrum resource utilization rate by providing dynamic spectrum access for second users (SUs) when the spectrum is used by primary users (PUs). When multiple SUs exist in a CRN at the same time, spectrum handoff will witness more complicated questions, such as non-stationary of environment and credit assignment. In this paper, we propose a multi-agent cooperative spectrum handoff method based on counterfactual multi-agent (COMA) algorithm to assist multiple SUs in handoff decision. With centralized training and decentralized execution, the proposed method can effectively deal with the problem of multi-agent spectrum Handoff. Simulation experiments prove that our method can guarantee the efficiency of handoff and quality of experience (QoE) requirement of SUs, and significantly improve the throughput of the system.
References
[1]
Abbas N, Nasser Y, El Ahmad K. Recent advances on artificial intelligence and learning techniques in cognitive radio networks[J]. EURASIP Journal on Wireless Communications and Networking, 2015, 2015(1): 1-20.
[2]
Kumar K, Prakash A, Tripathi R. Spectrum handoff in cognitive radio networks: A classification and comprehensive survey[J]. Journal of Network and Computer Applications, 2016, 61: 161-188.
[3]
Masuku D Z, Velempini M, Gasela N. A survey on spectrum handoff algorithm for continuous connectivity[C]//2018 International Conference on Intelligent and Innovative Computing Applications (ICONIC). IEEE, 2018: 1-5.
[4]
Wang C W, Wang L C, Adachi F. Modeling and analysis for reactive-decision spectrum handoff in cognitive radio networks[C]//2010 IEEE Global Telecommunications Conference GLOBECOM 2010. IEEE, 2010: 1-6.
[5]
Wang C W, Wang L C. Analysis of reactive spectrum handoff in cognitive radio networks[J]. IEEE journal on selected areas in communications, 2012, 30(10): 2016-2028.
[6]
Wang L C, Wang C W. Modeling and Analysis for Proactive-Decision Spectrum Handoff in Cognitive Radio Networks. 2009 IEEE International Conference on Communications, 1-6.
[7]
Wang L C, Wang C W, Chang C J. Modeling and analysis for spectrum handoffs in cognitive radio networks[J]. IEEE Transactions on Mobile Computing, 2011, 11(9): 1499-1513.
[8]
Zhu P, Li J, Han K, A new channel parameter for cognitive radio[C]//2007 2nd International Conference on Cognitive Radio Oriented Wireless Networks and Communications. IEEE, 2007: 482-486.
[9]
Heo J, Shin J, Nam J, Mathematical analysis of secondary user traffic in cognitive radio system[C]//2008 IEEE 68th Vehicular Technology Conference. IEEE, 2008: 1-5.
[10]
Wu Y, Hu F, Kumar S, Spectrum handoffs with mixed-priority queueing model over cognitive radio networks[C]//2013 IEEE Global Conference on Signal and Information Processing. IEEE, 2013: 1194-1197.
[11]
Song Y, Xie J. ProSpect: A proactive spectrum handoff framework for cognitive radio ad hoc networks without common control channel[J]. IEEE Transactions on Mobile Computing, 2012, 11(7): 1127-1139.
[12]
Wang L C, Wang C W, Adachi F. Load-balancing spectrum decision for cognitive radio networks[J]. IEEE Journal on Selected Areas in Communications, 2011, 29(4): 757-769.
[13]
Yin L, Li J, Yin S, Cognitive spectrum handoff game in the TV white space with radio environment map[C]//2012 IEEE 14th International Conference on Communication Technology. IEEE, 2012: 369-373.
[14]
Aghajani H, Ghasemi A. A congestion-game based scheme for handoff management in cognitive radio networks[C]//2013 21st Iranian Conference on Electrical Engineering (ICEE). IEEE, 2013: 1-6.
[15]
Liang Y C. Dynamic spectrum management: from cognitive radio to blockchain and artificial intelligence[M]. Springer Nature, 2020.
[16]
Bkassiny M, Li Y, Jayaweera S K. A survey on machine-learning techniques in cognitive radios[J]. IEEE Communications Surveys & Tutorials, 2012, 15(3): 1136-1159.
[17]
Zhou X, Sun M, Li G Y, Intelligent wireless communications enabled by cognitive radio and machine learning[J]. China Communications, 2018, 15(12): 16-48.
[18]
Hossain M A, Noor R M, Yau K L A, Comprehensive survey of machine learning approaches in cognitive radio-based vehicular ad hoc networks[J]. IEEE Access, 2020, 8: 78054-78108.
[19]
Guo J, Ji H, Li Y, A novel spectrum handoff management scheme based on SVM in cognitive radio networks[C]//2011 6th International ICST Conference on Communications and Networking in China (CHINACOM). IEEE, 2011: 645-649.
[20]
Çalhan A, Çeken C. Artificial neural network based vertical handoff algorithm for reducing handoff latency[J]. Wireless personal communications, 2013, 71(4): 2399-2415.
[21]
Zineb A B, Ayadi M, Tabbane S. QoE-based vertical handover decision management for cognitive networks using ANN[C]//2017 Sixth International Conference on Communications and Networking (ComNet). IEEE, 2017: 1-7.
[22]
Tabrizi H, Farhadi G, Cioffi J. Dynamic handoff decision in heterogeneous wireless systems: Q-learning approach[C]//2012 IEEE international conference on communications (ICC). IEEE, 2012: 3217-3222.
[23]
Wu Y, Hu F, Kumar S, A learning-based QoE-driven spectrum handoff scheme for multimedia transmissions over cognitive radio networks[J]. IEEE Journal on Selected Areas in Communications, 2014, 32(11): 2134-2148.
[24]
Xu N, Zhang H, Xu F, Q-learning based interference-aware channel handoff for partially observable cognitive radio ad hoc networks[J]. Chinese Journal of Electronics, 2017, 26(4): 856-863.
[25]
Busoniu L, Babuska R, De Schutter B. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2008, 38(2): 156-172.
[26]
Bowling M, Veloso M. Multiagent learning using a variable learning rate[J]. Artificial Intelligence, 2002, 136(2): 215-250.
[27]
Littman M L. Markov games as a framework for multi-agent reinforcement learning[M]//Machine learning proceedings 1994. Morgan Kaufmann, 1994: 157-163.
[28]
Hu J, Wellman M P. Multiagent reinforcement learning: theoretical framework and an algorithm[C]//ICML. 1998, 98: 242-250.
[29]
Abdallah S, Lesser V. A multiagent reinforcement learning algorithm with non-linear dynamics[J]. Journal of Artificial Intelligence Research, 2008, 33: 521-549.
[30]
Awheda M D, Schwartz H M. Exponential moving average Q-learning algorithm [C]//2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). IEEE, 2013: 31-38.
[31]
SUN Yu, CAO Lei, CHEN Xiliang, Overview of multi-agent deep reinforcement learning. Computer Engineering and Applications, 2020, 56)5(:13-24.
[32]
Buşoniu L, Babuška R, De Schutter B. Multi-agent reinforcement learning: An overview[J]. Innovations in multi-agent systems and applications-1, 2010: 183-221.
[33]
Sunehag P, Lever G, Gruslys A, Value-decomposition networks for cooperative multi-agent learning[J]. arXiv preprint arXiv:1706.05296, 2017.
[34]
Rashid T, Samvelyan M, Schroeder C, Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning[C]//International Conference on Machine Learning. PMLR, 2018: 4295-4304.
[35]
Hostallero W J K D E, Son K, Kim D, Learning to factorize with transformation for cooperative multi-agent reinforcement learning[C]//Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR. 2019.
[36]
Foerster J, Farquhar G, Afouras T, Counterfactual multi-agent policy gradients[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1).
[37]
Yu C, Velu A, Vinitsky E, The surprising effectiveness of mappo in cooperative, multi-agent games[J]. arXiv preprint arXiv:2103.01955, 2021.
[38]
Kraemer L, Banerjee B. Multi-agent reinforcement learning as a rehearsal for decentralized planning[J]. Neurocomputing, 2016, 190: 82-94.
[39]
Harati A, Ahmadabadi M N, Araabi B N. Knowledge-based multiagent credit assignment: A study on task type and critic information[J]. IEEE systems journal, 2007, 1(1): 55-67.
[40]
Naparstek O, Cohen K. Deep multi-user reinforcement learning for distributed dynamic spectrum access[J]. IEEE Transactions on Wireless Communications, 2018, 18(1): 310-323.
[41]
Li Shufeng, Shao Wei, Xie Ran, Yu Yujiang. Research on spectrum handoff problem based on Proximal Policy Optimization [J]. Communications Technology, 2021, 54(08):1917-1924.
Recommendations
Spectrum handoff in cognitive radio networks
The rapid progression in wireless networking has raised the demand of extra spectrum bands. The current fixed spectrum allocation policy used by the government agencies is unable to accommodate the growing demand of the wireless networks. The research ...
Performance analysis of spectrum handoff under heterogeneous spectrum environment in ad hoc and centralized CR networks
AbstractThe prime objective of cognitive radio (CR) is to address the problem of spectrum scarcity and inefficient usage of spectrum, by allowing the unlicensed or cognitive users (CUs) to access the underutilized licensed spectrum of the ...
Stochastic spectrum handoff protocols for partially observable cognitive radio networks
Recent studies have been conducted to indicate the ineffective usage of licensed bands due to static spectrum allocation. In order to improve spectrum utilization, cognitive radio (CR) is therefore suggested to dynamically exploit the opportunistic ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2021
146 pages
ISBN:9781450385848
DOI:10.1145/3510513
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 May 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICNCC 2021
ICNCC 2021: 2021 The 10th International Conference on Networks, Communication and Computing
December 10 - 12, 2021
Beijing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 22Total Downloads
- Downloads (Last 12 months)10
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format