Abstract
Cognitive radio is an emerging technology that is considered to be an evolution for software device radio in which cognition and decision-making components are included. The main function of cognitive radio is to exploit “spectrum holes” or “white spaces” to address the challenge of the low utilization of radio resources. Dynamic spectrum allocation, whose significant functions are to ensure that cognitive users access the available frequency and bandwidth to communicate in an opportunistic manner and to minimize the interference between primary and secondary users, is a key mechanism in cognitive radio networks. Reinforcement learning, which rapidly analyzes the amount of data in a model-free manner, dramatically facilitates the performance of dynamic spectrum allocation in real application scenarios. This paper presents a survey on the state-of-the-art spectrum allocation algorithms based on reinforcement learning techniques in cognitive radio networks. The advantages and disadvantages of each algorithm are analyzed in their specific practical application scenarios. Finally, we discuss open issues in dynamic spectrum allocation that can be topics of future research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ahmad A, Ahmad S, Rehmani MH, Hassan NU (2015) A survey on radio resource allocation in cognitive radio sensor networks. IEEE Commun Surv Tutor 17(2):888–917
Ahmed E, Gani A, Abolfazli S, Yao LJ, Khan SU (2014) Channel assignment algorithms in cognitive radio networks: taxonomy, open issues, and challenges. IEEE Commun Surv Tutor 18(1):795–823
Al-Rawi HA, Ng MA, Yau KLA (2015) Application of reinforcement learning to routing in distributed wireless networks: a review. Artif Intell Rev 43(3):381–416
Alsarhan A, Agarwal A (2011) Profit optimization in multi-service cognitive mesh network using machine learning. EURASIP J Wirel Commun Netw 1:36
Anifantis E, Karyotis V, Papavassiliou S (2012) A markov random field framework for channel assignment in cognitive radio networks. In: 2012 IEEE international conference on pervasive computing and communications workshops (PERCOM workshops). IEEE, pp 770–775
Berthold U, Fu F, Van M der Schaar, Jondral FK (2008) Detection of spectral resources in cognitive radios using reinforcement learning. In: New frontiers in dynamic spectrum access networks, pp 1–5
Bkassiny M, Li Y, Jayaweera SK (2013) A survey on machine-learning techniques in cognitive radios. IEEE Commun Surv Tutor 15(3):1136–1159
Busoniu L, Babuska R, De Schutter B (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C Appl Rev 38(2):156–172
Chen S, Huang Y, Namuduri K (2011) A factor graph based dynamic spectrum allocation approach for cognitive network. In: Wireless communications and networking conference (WCNC), 2011 IEEE. IEEE, pp 850–855
Cheng X, Jiang M (2011) Cognitive radio spectrum assignment based on artificial bee colony algorithm. In: 2011 IEEE 13th international conference on communication technology (ICCT). IEEE, pp 161–164
Faganello LR, Kunst R, Both CB, Granville LZ (2013) Improving reinforcement learning algorithms for dynamic spectrum allocation in cognitive sensor networks. In: Wireless communications and networking conference, pp 35–40
Feng Z, Liang L, Tan L, Zhang P (2009) Q-learning based heterogenous network self-optimization for reconfigurable network with CPC assistance. Sci China (Ser F) 52(12):2360–2368
Han G, Xiao L, Poor HV (2017) Two-dimensional anti-jamming communication based on deep reinforcement learning. In: IEEE international conference on acoustics, speech and signal processing, pp 2087–2091
Hossain E, Bhargava V (2007) Cognitive wireless communication networks. Springer, New York
Iii JM (2000) Cognitive radio: an integrated agent architecture for software defined radio, Ph.D. dissertation. ResearchGate 6(4):13–18
Le HST, Ly HD (2008) Opportunistic spectrum access using fuzzy logic for cognitive radio networks. In: Second international conference on communications and electronics, ICCE 2008. IEEE, pp 240–245
Levorato M, Firouzabadi S, Goldsmith A (2012) A learning framework for cognitive interference networks with partial and noisy observations. IEEE Trans Wirel Commun 11(9):3101–3111
Li H, Zhu G, Liang Z, Chen Y (2010) A survey on distributed opportunity spectrum access in cognitive network. In: International conference on wireless communications networking and mobile computing, pp 1–4
Li Y, Feng Z, Chen S, Chen Y, Xu D, Zhang P, Zhang Q (2011) Radio resource management for public femtocell networks. EURASIP J Wirel Commun Netw 1:181
Lilith N, Dogancay K (2005) Distributed reduced-state sarsa algorithm for dynamic channel allocation in cellular networks featuring traffic mobility. IEEE Int Conf Commun 2:860–865
Lv C, Wang J, Yu F, Dai H (2013) A Q-learning-based dynamic spectrum allocation algorithm. ICCSEE-13
Marinho J, Monteiro E (2012) Cognitive radio: survey on communication protocols, spectrum decision issues, and future research directions. Wirel Netw 18(2):147–164
Mitola J, Maguire GQ (1999) Cognitive radio: making software radios more personal. IEEE Pers Commun 6(4):13–18
Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. Comput Sci
Nie J, Haykin S (1997) A Q-learning-based dynamic channel assignment technique for mobile communication systems. IEEE Trans Veh Technol 48(5):1676–1687
Qadir J (2016) Artificial intelligence based cognitive routing for cognitive radio networks. Artif Intell Rev 45(1):25–96
Ru M, Yin S, Qu Z (2017) Power and spectrum allocation in d2d networks based on coloring and chaos genetic algorithm. Procedia Comput Sci 107:183–189
Salameh HAB (2011) Throughput-oriented channel assignment for opportunistic spectrum access networks. Math Comput Modell 53(11–12):2108–2118
Shu T, Krunz M (2010) Exploiting microscopic spectrum opportunities in cognitive radio networks via coordinated channel access. IEEE Trans Mob Comput 9(11):1522–1534
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, vol 1. MIT Press, Cambridge
Tanwongvarl C, Chantaraskul S (2015) Performance comparison of learning techniques for intelligent channel assignment in cognitive wireless sensor networks. In: Seventh international conference on ubiquitous and future networks, pp 503–507
Teng Y, Zhang Y, Niu F, Dai C (2010) Reinforcement learning based auction algorithm for dynamic spectrum access in cognitive radio networks. In: Vehicular technology conference fall, pp 1–5
Teng Y, Yu FR, Han K, Wei Y, Zhang Y (2013) Reinforcement-learning-based double auction design for dynamic spectrum access in cognitive radio networks. Wirel Pers Commun 69(2):771–791
Tragos EZ, Zeadally S, Fragkiadakis AG, Siris VA (2013) Spectrum assignment in cognitive radio networks: a comprehensive survey. IEEE Commun Surv Tutor 15(3):1108–1135
Wang W, Kwasinski A, Niyato D, Han Z (2016) A survey on applications of model-free strategy learning in cognitive wireless networks. IEEE Commun Surv Tutor 18(3):1717–1757
Xiao L, Li Y, Dai C, Dai H, Poor HV (2017) Reinforcement learning-based NOMA power allocation in the presence of smart jamming. IEEE Trans Veh Technol 67:3377–3389
Yang M, Grace D (2011) Cognitive radio with reinforcement learning applied to multicast downlink transmission with power adjustment. Wirel Pers Commun 57(1):73–87
Yang R, Ye F, et al (2010) Non-cooperative spectrum allocation based on game theory in cognitive radio networks. In: 2010 IEEE fifth international conference on bio-inspired computing: theories and applications (BIC-TA). IEEE, pp 1134–1137
Yau KLA, Komisarczuk P, Teal PD (2012) Reinforcement learning for context awareness and intelligence in wireless networks: review, new features and open issues. J Netw Comput Appl 35(1):253–267
Yi L, Hong J (2012) Q-learning for dynamic channel assignment in cognitive wireless local area network with fibre-connected distributed antennas. J China Univer Posts Telecommun 19(4):80–85
Yu L, Liu C, Liu Z, Hu W (2010) Heuristic spectrum assignment algorithm in distributed cognitive networks. In: 2010 6th International conference on wireless communications networking and mobile computing (WiCOM). IEEE, pp 1–5
Zhang Y, Lee C, Niyato D, Wang P (2013) Auction approaches for resource allocation in wireless systems: a survey. IEEE Commun Surv Tutor 15(3):1020–1041
Zhao C, Zou M, Shen B, Kim B, Kwak K (2008) Cooperative spectrum allocation in centralized cognitive networks using bipartite matching. In: Global telecommunications conference, IEEE GLOBECOM 2008. IEEE, pp 1–6
Zhao Q, Sadler BM (2007) A survey of dynamic spectrum access. IEEE Signal Process Mag 24(3):79–89
Acknowledgements
This work was supported in part by special funds from the central finance to support the development of local universities under No. 400170044, the project supported by the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences under Grant No. 20180106, the science and technology program of Guangdong Province under Grant No. 2016B090918031, the degree and graduate education reform project of Guangdong Province under Grant No. 2016JGXM_MS_26, the foundation of key laboratory of machine intelligence and advanced computing of the Ministry of Education under Grant No. MSC-201706A and the higher education quality projects of Guangdong Province and Guangdong University of Technology.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, Y., Ye, Z., Wan, P. et al. A survey of dynamic spectrum allocation based on reinforcement learning algorithms in cognitive radio networks. Artif Intell Rev 51, 493–506 (2019). https://doi.org/10.1007/s10462-018-9639-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-018-9639-x