Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

Towards Intelligent Adaptive Edge Caching Using Deep Reinforcement Learning

Published: 01 February 2024 Publication History

Abstract

The tremendous expansion of edge data traffic poses great challenges to network bandwidth and service responsiveness for mobile computing. Edge caching has emerged as a promising method to alleviate these issues by storing a portion of data at the network edge. However, existing caching approaches suffer from either poor caching efficiency with low content-hit ratio or unintelligence of caching policies lacking self-adjustability. In this article, we propose ICE, a novel Intelligent Edge Caching scheme using a deep reinforcement learning (DRL) method to capture specific valuable information from the requested data. With the benefit of our proposed popularity model based on Newton's law of cooling, ICE fully takes into account the popularity of the contents to be cached and leverages the formulated Markov decision model to decide whether or not the contents should be cached. Moreover, to further improve the caching efficiency, we propose a novel distributed multi-node caching framework, named DCCC, assisted by a multi-tiered caching hierarchy. Comprehensive experiments show that the single-node ICE scheme greatly improves the cache hit rate and contents exchanging time in comparison with both DRL-based and legacy approaches, and our distributed multi-node caching scheme DCCC further significantly improves the overall utilization of caching space.

References

[1]
T. Wang, J. Mao, M. Chen, G. Liu, J. Di, and S. Yu, “ICE: Intelligent caching at the edge,” in Proc. IEEE Glob. Commun. Conf., 2021, pp. 1–6.
[2]
L. Wang, Y. L. Che, J. Long, L. Duan, and K. Wu, “Multiple access mmWave design for UAV-aided 5G communications,” IEEE Wirel. Commun., vol. 26, no. 1, pp. 64–71, Feb. 2019.
[3]
A. Tian et al., “Efficient federated DRL-based cooperative caching for mobile edge networks,” IEEE Trans. Netw. Service Manag., vol. 20, no. 1, pp. 246–260, Mar. 2023.
[4]
Z. Yu, J. Hu, G. Min, Z. Zhao, W. Miao, and M. S. Hossain, “Mobility-aware proactive edge caching for connected vehicles using federated learning,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 8, pp. 5341–5351, Aug. 2021.
[5]
D. R.-J. G.-J. Rydning, R. John, and G. John, “The digitization of the world from edge to core,” Framingham: Int. Data Corporation, vol. 16, pp. 1–28, 2018.
[6]
C. Stergiou, K. E. Psannis, B. G. Kim, and B. Gupta, “Secure integration of IoT and cloud computing,” Future Gener. Comput. Syst., vol. 78, no. PT.3, pp. 964–975, 2016.
[7]
B. Varghese and R. Buyya, “Next generation cloud computing: New trends and research directions,” Future Gener. Comput. Syst., vol. 79, pp. 849–861, 2018.
[8]
A. Abouaomar, A. Filali, and A. Kobbane, “Caching, device-to-device and fog computing in 5th cellular networks generation : Survey,” in Proc. Int. Conf. Wirel. Netw. Mobile Commun., 2017, pp. 1–6.
[9]
Soubhik and Chakraborty, “Algorithmic nuggets in content delivery,” Comput. Rev., vol. 57, no. 2, pp. 103–103, 2016.
[10]
S. Wang, X. Zhang, Y. Zhang, L. Wang, J. Yang, and W. Wang, “A survey on mobile edge networks: Convergence of computing, caching and communications,” IEEE Access, vol. 5, pp. 6757–6779, 2017.
[11]
Z. Su, Q. Xu, F. Hou, Q. Yang, and Q. Qi, “Edge caching for layered video contents in mobile social networks,” IEEE Trans. Multimedia, vol. 19, no. 10, pp. 2210–2221, Oct. 2017.
[12]
K. Geetha and N. A. Gounden, “Dynamic semantic LFU policy with victim tracer (DSLV): A customizing technique for client cache,” Arabian J. Sci. Eng., vol. 42, pp. 725–737, 2017.
[13]
T. G. Hendrantoro and A. Affandi, “Early result from adaptive combination of LRU, LFU and FIFO to improve cache server performance in telecommunication network,” in Proc. Int. Seminar Intell. Technol. Appl., 2015, pp. 429–432.
[14]
B. Jiang, P. Nain, and D. Towsley, “LRU cache under stationary requests,” ACM SIGMETRICS Perform. Eval. Rev., vol. 45, no. 2, pp. 24–26, 2017.
[15]
M. C. Gursoy, C. Zhong, and S. Velipasalar, “Deep multi-agent reinforcement learning for cooperative edge caching,” Mach. Learn. Future Wirel. Commun., John Wiley & Sons, Ltd., ch. 21, pp. 439–457, 2020. [Online]. Available: https://doi.org/10.1002/9781119562306.ch21
[16]
D. Qiao, S. Guo, D. Liu, S. Long, P. Zhou, and Z. Li, “Adaptive federated deep reinforcement learning for proactive content caching in edge computing,” IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 12, pp. 4767–4782, Dec. 2022.
[17]
Q. Fan, X. Li, J. Li, Q. He, K. Wang, and J. Wen, “PA-cache: Evolving learning-based popularity-aware content caching in edge networks,” IEEE Trans. Netw. Service Manag., vol. 18, no. 2, pp. 1746–1757, Jun. 2021.
[18]
N. Nomikos, S. Zoupanos, T. Charalambous, and I. Krikidis, “A survey on reinforcement learning-aided caching in heterogeneous mobile edge networks,” IEEE Access, vol. 10, pp. 4380–4413, 2022.
[19]
W.-X. Liu, J. Zhang, Z.-W. Liang, L.-X. Peng, and J. Cai, “Content popularity prediction and caching for ICN: A deep learning approach with sdn,” IEEE Access, vol. 6, pp. 5075–5089, 2017.
[20]
Y. Im, P. Prahladan, T. H. Kim, Y. G. Hong, and S. Ha, “SNN-cache: A practical machine learning-based caching system utilizing the inter-relationships of requests,” in Proc. 52nd Annu. Conf. Inf. Sci. Syst., 2018, pp. 1–6.
[21]
K. C. Tsai, L. Wang, and Z. Han, “Mobile social media networks caching with convolutional neural network,” in Proc. IEEE Wirel. Commun. Netw. Conf. Workshops, 2018, pp. 83–88.
[22]
A. Lekharu, M. Jain, A. Sur, and A. Sarkar, “Deep learning model for content aware caching at MEC servers,” IEEE Trans. Netw. Service Manag., vol. 19, no. 2, pp. 1413–1425, Jun. 2022.
[23]
Y. Zhang, Y. Li, R. Wang, J. Lu, X. Ma, and M. Qiu, “PSAC: Proactive sequence-aware content caching via deep learning at the network edge,” IEEE Trans. Netw. Sci. Eng., vol. 7, no. 4, pp. 2145–2154, Fourth Quarter 2020.
[24]
N. Kumar, S. N. Swain, and C. Siva Ram Murthy, “A novel distributed Q-learning based resource reservation framework for facilitating D2D content access requests in LTE-a networks,” IEEE Trans. Netw. Service Manag., vol. 15, no. 2, pp. 718–731, Jun. 2018.
[25]
S. Liu, C. Zheng, Y. Huang, and T. Q. S. Quek, “Distributed reinforcement learning for privacy-preserving dynamic edge caching,” IEEE J. Sel. Areas Commun., vol. 40, no. 3, pp. 749–760, Mar. 2022.
[26]
H. Zheng, H. Zhou, N. Wang, P. Chen, and S. Xu, “Reinforcement learning for energy-efficient edge caching in mobile edge networks,” in Proc. IEEE Conf. Comput. Commun. Workshops, 2021, pp. 1–6.
[27]
R. Wang, Z. Kan, Y. Cui, D. Wu, and Y. Zhen, “Cooperative caching strategy with content request prediction in Internet of Vehicles,” IEEE Internet Things J., vol. 8, no. 11, pp. 8964–8975, Jun. 2021.
[28]
C. Zhong, M. C. Gursoy, and S. Velipasalar, “Deep reinforcement learning-based edge caching in wireless networks,” IEEE Trans. Cogn. Commun. Netw., vol. 6, no. 1, pp. 48–61, Mar. 2020.
[29]
L. T. Tan and R. Q. Hu, “Mobility-aware edge caching and computing in vehicle networks: A deep reinforcement learning,” IEEE Trans. Veh. Technol, vol. 67, no. 11, pp. 10 190–10 203, Nov. 2018.
[30]
Z. Song et al., “Learning relaxed belady for content distribution network caching,” in Proc. 17th USENIX Symp. Netw. Syst. Des. Implementation, 2020, pp. 529–544.
[31]
J. Yan, Y. Jiang, F. Zheng, F. R. Yu, and X. You, “Distributed edge caching with content recommendation in fog-RANs via deep reinforcement learning,” in Proc. IEEE Int. Conf. Commun. Workshops, 2020, pp. 1–6.
[32]
S. Zhang, P. He, K. Suto, P. Yang, L. Zhao, and X. Shen, “Cooperative edge caching in user-centric clustered mobile networks,” IEEE Trans. Mobile Comput., vol. 17, no. 8, pp. 1791–1805, Aug. 2018.
[33]
Y. M. Saputra, D. T. Hoang, D. N. Nguyen, E. Dutkiewicz, D. Niyato, and D. I. Kim, “Distributed deep learning at the edge: A novel proactive and cooperative caching framework for mobile edge networks,” IEEE Wireless Commun. Lett., vol. 8, no. 4, pp. 1220–1223, Aug. 2019.
[34]
S. Chen, Z. Yao, X. Jiang, J. Yang, and L. Hanzo, “Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks,” IEEE Trans. Commun., vol. 69, no. 4, pp. 2441–2456, Apr. 2021.
[35]
X. Xia, F. Chen, Q. He, J. Grundy, M. Abdelrazek, and H. Jin, “Online collaborative data caching in edge computing,” IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 02, pp. 281–294, Feb. 2021.
[36]
G. Zhao, H. Xu, Y. Zhao, C. Qiao, and L. Huang, “Offloading tasks with dependency and service caching in mobile edge computing,” IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 11, pp. 2777–2792, Nov. 2021.
[37]
H. Wang, G. Tang, K. Wu, and J. Wang, “PLVER: Joint stable allocation and content replication for edge-assisted live video delivery,” IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 01, pp. 218–230, Jan. 2022.
[38]
X. Wang, C. Wang, X. Li, V. C. M. Leung, and T. Taleb, “Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching,” IEEE Internet Things J., vol. 7, no. 10, pp. 9441–9455, Oct. 2020.
[39]
Z. Zhao, W. Zhou, D. Dan, J. Xia, and L. Fan, “Intelligent mobile edge computing with pricing in Internet of Things,” IEEE Access, vol. 8, pp. 37727–37735, 2020.
[40]
Google, Tensorflow1.9.0, 2018. [Online]. Available: https://github.com/tensorflow/tensorflow/releases/tag/v1.9.0
[41]
R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” Adv. Neural Inf. Process. Syst., vol. 12, pp. 1057–1063, 1999.

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Mobile Computing
IEEE Transactions on Mobile Computing  Volume 23, Issue 10
Oct. 2024
1160 pages

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 February 2024

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 31 Dec 2024

Other Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media