research-article

FAS-DQN: Freshness-Aware Scheduling via Reinforcement Learning for Latency-Sensitive Applications

Authors:

Bing GuoAuthors Info & Claims

IEEE Transactions on Computers, Volume 71, Issue 10

Pages 2381 - 2394

https://doi.org/10.1109/TC.2021.3129342

Published: 01 October 2022 Publication History

Abstract

The demand for real-time data processing has become increasingly attractive in Cyber-Physical Systems(CPSs), especially for data-intensive embedded real-time applications. In order to timely perceive and respond to environmental changes, the basic design requirement in such systems is to provide data service with high freshness. As modern CPSs become more complex, there are a broad set of system mode switch behaviors, some unforeseen, in a dynamic computational environment. However, conventional control algorithms can hardly handle such new scenarios, since most of them assume that the operational behavior is fixed. In this paper, we study the problem of how to maximize the freshness of data in multi-modal systems. We first use a recently proposed new conception, namely Age of Information (AoI) to quantify the freshness of data by combining the AoI metric with real-time constraints. Then, we propose, to our knowledge, the first freshness-aware scheduling solution to settle the problem via deep reinforcement learning(RL). To be specific, we develop an RL framework that can continuously update its scheduling strategies and maximize the freshness of data in the long term. Extensive simulation experiments are conducted and the results demonstrate that the proposed FAS-DQN outperforms other traditional state-of-the-art methods in terms of data freshness.

References

[1]

T. Abdelzaheret al., “Decision-driven execution: A distributed resource management paradigm for the age of IoT,” in Proc. IEEE 37th Int. Conf. Distrib. Comput. Syst., 2017, pp. 1825–1835.

[2]

A. Asl, A. Ar, A. Deq, A. Hk, and S. B. Ling, “Deep reinforcement learning for wireless sensor scheduling in cyber physical systems,” Automatica, vol. 113, 2019, Art. no.

[3]

S. K. Baruah, L. E. Rosier, and R. R. Howell, “Algorithms and complexity concerning the preemptive scheduling of periodic, real-time tasks on one processor,” Real-Time Syst., vol. 2, no. 4, pp. 301–324, 1990.

Digital Library

[4]

S. Bateni and C. Liu, “ApNet: Approximation-aware real-time neural network,” in Proc. IEEE Real-Time Syst. Symp., 2018, pp. 67–79.

[5]

S. Bateni, H. Zhou, Y. Zhu, and C. Liu, “Predjoule: A timing-predictable energy optimization framework for deep neural networks,” in Proc. IEEE Real-Time Syst. Symp., 2018, pp. 107–118.

[6]

A. S. Leong, S. Dey and D. E. Quevedo, “Sensor scheduling in variance based event triggered estimation with packet drops,” IEEE Trans. Autom. Control, vol. 62, no. 5, pp. 1880–1895, Apr. 2017.

[7]

P. Duda, L. Rutkowski, M. Jaworski, and D. Rutkowska, “On the parzen kernel-based probability density function learning procedures over time-varying streaming data with applications to pattern classification,” IEEE Trans. Cybern., vol. 50, no. 4, pp. 1683–1696, Apr. 2020.

[8]

S. Emiliano, S. Abusayeed, H. Song, J. Ulf, and G. Mikael, “Industrial internet of things: Challenges, opportunities, and directions,” IEEE Trans. Ind. Inf., vol. 14, no. 11, 4724–4734, Nov. 2018.

[9]

Z. Guo, K. Yang, S. Vaidhun, S. Arefin, and H. Xiong, “Uniprocessor mixed-criticality scheduling with graceful degradation by completion rate,” in Proc. IEEE Real-Time Syst. Symp., 2018, pp. 373–383.

[10]

S. Han, D. Chen, M. Xiong, K. Lam, A. K. Mok, and K. Ramamritham, “Schedulability analysis of deferrable scheduling algorithms for maintaining real-time data freshness,” IEEE Trans. Comput., vol. 63, no. 4, pp. 979–994, Apr. 2014.

Digital Library

[11]

S. Hanet al., “Online mode switch algorithms for maintaining data freshness in dynamic cyber-physical systems,” IEEE Trans. Knowl. Data Eng., vol. 28, no. 3, pp. 756–769, Mar. 2016.

[12]

R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” IEEE Trans. Neural Netw., vol. 9, no. 5, p. 1054, 1998.

Digital Library

[13]

S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?,” in Proc. INFOCOM, 2012, pp. 2731–2735.

[14]

J. E. Kim, T. Abdelzaher, L. Sha, A. Bar-Noy, and R. Hobbs, “Sporadic decision-centric data scheduling with normally-off sensors,” in Proc. Real-Time Syst. Symp., 2016, pp. 135–145.

[15]

T.-W. Kuo and S.-J. Ho, “Similarity-based load adjustment for static real-time transaction systems,” IEEE Trans. Comput., vol. 49, no. 2, pp. 112–126, Feb. 2000.

Digital Library

[16]

K.-Y. Lam, N. W.-H. Tsang, S. Han, W. Zhang, J. K.-Y. Ng, and A. Nath, “Activity tracking and monitoring of patients with alzheimer's disease,” Multimedia Tools Appl., vol. 76, no. 1, pp. 489–521, 2017.

Digital Library

[17]

G. Li, C. Zhou, J. Li, and B. Guo, “Maintaining data freshness in distributed cyber-physical systems,” IEEE Trans. Comput., vol. 68, no. 7, pp. 1077–1090, Jul. 2019.

[18]

J. Li, M. Xiong, V. Lee, L. Shu, and G. Li, “Workload-efficient deadline and period assignment for maintaining temporal consistency under EDF,” IEEE Trans. Comput., vol. 62, no. 6, pp. 1255–1268, Jun. 2013.

Digital Library

[19]

D. Liuet al., “Scheduling analysis of imprecise mixed-criticality real-time tasks,” IEEE Trans. Comput., vol. 67, no. 7, pp. 975–991, Jul. 2018.

Digital Library

[20]

F. S. Melo and M. I. Ribeiro, “Convergence of Q-learning with linear function approximation,” in Proc. Control Conf., 2007, pp. 2671–2678.

[21]

V. Mnihet al., “Playing atari with deep reinforcement learning,” 2013,.

[22]

V. Mnihet al., “Human-level control through deep reinforcement learning,” Nature, vol. 18, no. 7540, 2015, Art. no.

[23]

Y. Mo, E. Garone, and B. Sinopoli, “On infinite horizon sensor scheduling,” Syst. Control Lett., vol. 67, pp. 65–70, 2014.

[24]

A. Redder, A. Ramaswamy, and D. E. Quevedo, “Deep reinforcement learning for scheduling in large-scale networked control systems,” 2019,.

[25]

B. Scholkopf, J. Platt, and T. Hofmann, “Greedy layer-wise training of deep networks,” in Proc. ACM/IEEE Int. Conf. Adv. Neural Inf. Process. Syst., 2007., pp. 31–40.

[26]

W. Shuang, X. Ren, D. Subhrakanti, and S. Ling, “Optimal scheduling of multiple sensors over shared channels with packet transmission constraint,” Automatica, vol. 96, pp. 22–31, 2018.

[27]

D. Silveret al., “Mastering the game of go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, 2016, Art. no.

[28]

Y. Sun and B. Cyr, “Sampling for data freshness optimization: Non-linear age functions,” J. Commun. Netw., vol. 21, no. 3, pp. 204–219, 2019.

[29]

M. Xiong and K. Ramamritham, “Deriving deadlines and periods for real-time update transactions,” IEEE Trans. Comput., vol. 53, no. 5, pp. 567–583, May 2004.

Digital Library

[30]

R. T. Y. Sun, I. Kadota and E. Modiano, “Age of information: A new metric for information freshness,” Synth. Lectures Commun. Netw., vol. 12, no. 2, pp. 1–224, 2019.

[31]

X. Yuan, B. Huang, Y. Wang, C. Yang, and W. Gui, “Deep learning-based feature representation and its application for soft sensor modeling with variable-wise weighted SAE,” IEEE Trans. Ind. Inf., vol. 14, no. 7, pp. 3235–3243, Jul. 2018.

[32]

F. Zhang and A. Burns, “Analysis of hierarchical EDF pre-emptive scheduling,” in Proc. IEEE Int. Real-Time Syst. Symp., 2007, pp. 423–434.

[33]

F. Zhang and A. Burns, “Schedulability analysis for real-time systems with EDF scheduling,” IEEE Trans. Comput., vol. 58, no. 9, pp. 1250–1258, Sep. 2009.

Digital Library

[34]

Q. Zhang, M. Lin, L. T. Yang, Z. Chen, and P. Li, “Energy-efficient scheduling for real-time systems based on deep Q-learning model,” IEEE Trans. Sustain. Comput., vol. 4, no. 1, pp. 132–141, Jan.–Mar. 2019.

[35]

L. Zhao, W. Zhang, J. Hu, A. Abate, and C. J. Tomlin, “On the optimal solutions of the infinite horizon linear sensor scheduling problem,” IEEE Trans. Autom. Control, vol. 59, no. 10, pp. 2825–2830, Oct. 2014.

[36]

Z. Zhou, C. Fu, C. Xue, and S. Han, “Transmit or discard: Optimizing data freshness in networked embedded systems with energy harvesting sources,” in Proc. 56th ACM/IEEE Des. Automat. Conf., 2019, pp. 1–6.

Cited By

Gong SCui LGu BLyu BHoang DNiyato D(2023)Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-Aided and Wireless-Powered Wireless NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2023.325972122:11(8114-8127)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TWC.2023.3259721

Index Terms

FAS-DQN: Freshness-Aware Scheduling via Reinforcement Learning for Latency-Sensitive Applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Deferrable Scheduling for Maintaining Real-Time Data Freshness: Algorithms, Analysis, and Results

The periodic update transaction model has been used to maintain freshness (or temporal validity) of real-time data. Period and deadline assignment has been the main focus in the past studies such as the More-Less scheme [25] in which update transactions ...
Deep Reinforcement Learning in VizDoom via DQN and Actor-Critic Agents
Advances in Computational Intelligence
Abstract
In this work, we study the problem of learning reinforcement learning-based agents in a first-person shooter environment VizDoom. We compare several well-known architectures, such as DQN, DDQN, A3C, and Curiosity-driven model, while highlighting ...
Information Freshness in Cache Updating Systems
We consider a cache updating system with a source, a cache and a user. There are <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> files. The source keeps the freshest version of the files which are updated with known rates <...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Computers

IEEE Transactions on Computers Volume 71, Issue 10

Oct. 2022

347 pages

ISSN:0018-9340

Issue’s Table of Contents

0018-9340 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 October 2022

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gong SCui LGu BLyu BHoang DNiyato D(2023)Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-Aided and Wireless-Powered Wireless NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2023.325972122:11(8114-8127)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TWC.2023.3259721

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents