Nothing Special   »   [go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

E-DBRL: efficient double broad reinforcement learning for adaptive traffic signal control

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Efficient traffic signal management is crucial for regulating traffic flow and fostering sustainable development within road transportation systems. To address the challenges in traffic management, numerous studies have applied the Adaptive Traffic Signal Control (ATSC) technology, using Deep Reinforcement Learning (DRL) to decrease vehicles’ average waiting times. Nonetheless, the intricate nature of DRL, characterized by its extensive parameter connections, often complicates the assurance of real-time responsiveness. Additionally, by prioritizing reduced waiting times, these methods may overlook potential rises in queue lengths, risking congestion. In this paper, we propose an Efficient Double Broad Reinforcement Learning (E-DBRL) algorithm based on a Double Broad Q-Network (Double BQN) to alleviate the overestimation of action values common in Broad Reinforcement Learning (BRL). To enhance the Quality of Experience (QoE) of drivers, we develop a new reward function that optimizes the average waiting time and the range between the longest and shortest waiting times, thus avoiding the need for dimension normalization. Moreover, we conduct simulation experiments using actual traffic data collected from Hangzhou, China. The experimental results indicate that, compared to the traditional Double DQN, the proposed E-DBRL algorithm achieves a 45.78% reduction in the average training time per round and a 5.57% increase in the average rewards.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets used during the cur-rent study are available on the GitHub platform link https://github.com/traffic-signal-control/sample-code/tree/master/data.

References

  1. Ji B et al (2021) A survey of computational intelligence for 6G: Key technologies, applications and trends. IEEE Trans Ind Inform 17(10):7145–7154

    Article  Google Scholar 

  2. Roy S, Basu D (2020) Selection of intervention areas for improving travel condition of walk-accessed bus users with a focus on their accessibility: An experience in Bhubaneswar. Transp Policy 96:29–39

    Article  Google Scholar 

  3. Zhang Y et al (2024) Safety-aware vehicle-following driving optimization of intelligent and connected vehicle at signalized road intersection. Control Eng Pract 142:105765

    Article  Google Scholar 

  4. Wei H et al (2021) Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation. ACM SIGKDD Explor Newsl 22(2):12–18

    Article  Google Scholar 

  5. Golden B, Wang X, Wasil E (2023) The evolution of the vehicle routing problem-a survey of VRP research and practice from 2005 to 2022. The evolution of the vehicle routing problem: a survey of VRP research and practice from 2005 to 2022. Cham, Springer Nature Switzerland, pp 1–64

  6. Noaeen M et al (2022) Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Syst Appl 199:116830

    Article  Google Scholar 

  7. Wei H et al (2019) Colight: Learning network-level cooperation for traffic signal control. In: Proceedings of the 28th ACM international conference on information and knowledge management

  8. Oroojlooy A et al (2020) Attendlight: Universal attention-based reinforcement learning model for traffic signal control. Adv Neural Inf Process Syst 33:4079–4090

    Google Scholar 

  9. Ye B-L et al (2019) A survey of model predictive control methods for traffic signal control. IEEE/CAA J Automatic Sin 6(3):623-640

    Article  MathSciNet  Google Scholar 

  10. Mnih V et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  11. Li X, Li J, Shi H (2023) A multi-agent reinforcement learning method with curriculum transfer for large-scale dynamic traffic signal control. Appl Intell, pp 1–15

  12. Genders W, Razavi S (2020) Policy analysis of adaptive traffic signal control using reinforcement learning. J Comput Civ Eng 34(1):04019046

    Article  Google Scholar 

  13. Li Dongdong, Dong Jiuxiang (2023) Fuzzy Control Based on Reinforcement Learning and Subsystem Error Derivatives for Strict-Feedback Systems With an Observer. IEEE Trans Fuzzy Syst 31(8):2509–2521

    Article  Google Scholar 

  14. Chen CLP, Liu Z (2017) Broad learning system: An effective and efficient incremental learning system without the need for deep architecture. IEEE Trans Neural Netw Learn Syst 29(1):10–24

    Article  MathSciNet  Google Scholar 

  15. Chang HH, Liu L, Yi Y (2020) Deep echo state Q-network (DEQN) and its application in dynamic spectrum sharing for 5G and beyond. IEEE Trans Neural Netw Learn Syst 33(3):929–939

    Article  Google Scholar 

  16. Wei X et al (2020) Broad reinforcement learning for supporting fast autonomous IoT. IEEE Internet Things J 7(8):7010–7020

    Article  Google Scholar 

  17. Liu D et al (2020) On training traffic predictors via broad learning structures: A benchmark study. IEEE Trans Syst Man Cybern: Syst 52(2):749–758

    Article  Google Scholar 

  18. Tang J et al (2020) Semi-supervised double duelling broad reinforcement learning in support of traffic service in smart cities. IET Intell Trans Syst 14(10):1278–1285

    Article  Google Scholar 

  19. Eom M, Kim B-I (2020) The traffic signal control problem for intersections: a review. Europe Transp Res Rev 12:1–20

    Google Scholar 

  20. Chen Y et al (2023) Traffic signal optimization control method based on adaptive weighted averaged double deep Q network. Appl Intell, pp 1–22

  21. Shen Z et al (2020) A novel learning method for multi-intersections aware traffic flow forecasting. Neurocomputing 398:477–484

    Article  Google Scholar 

  22. Kartikasari RY, Prakarsa G, Pradeka D (2020) Optimization of traffic light control using fuzzy logic sugeno method. InT J Glob Oper Res 1(2):51–61

    Google Scholar 

  23. Li D, Dong J (2024) Fuzzy weight-based reinforcement learning for event-triggered optimal backstepping control of fractional-order nonlinear systems. IEEE Trans Fuzzy Syst 32(1):1–12

    Article  Google Scholar 

  24. Zhu Z et al (2023) Transfer Learning in Deep Reinforcement Learning: A Survey. IEEE Tran Pattern Anal Mach Intell PP.11:1–20

  25. Li D, Dong J (2023) Output-feedback optimized consensus for directed graph multi-agent systems based on reinforcement learning and subsystem error derivatives. Inf Sci 649:119577

    Article  Google Scholar 

  26. Gong X et al (2021) Research review for broad learning system: Algorithms, theory, and applications. IEEE Trans Cybern 52(9):8922–8950

    Article  Google Scholar 

  27. Zhao H et al (2020) Semi-supervised broad learning system based on manifold regularization and broad network. IEEE Trans Circ Syst I: Regular Pap 67(3):983–994

    MathSciNet  Google Scholar 

  28. Zhang L et al (2020) Analysis and variants of broad learning system. IEEE Trans Syst Man Cybern: Syst 52(1):334–344

    Article  Google Scholar 

  29. Zhang C et al (2019) Deep transfer learning for intelligent cellular traffic prediction based on cross-domain big data. IEEE J Sel Areas Commun 37(6):1389–1401

    Article  Google Scholar 

  30. Guo W, Chen S, Yuan X (2023) H-BLS: a hierarchical broad learning system with deep and sparse feature learning. Appl Intell 53(1):153–168

    Article  Google Scholar 

  31. Ali R et al (2020) Optic disk and cup segmentation through fuzzy broad learning system for glaucoma screening. IEEE Trans Ind Inform 17(4):2476–2487

    Article  Google Scholar 

  32. Wang H et al (2020) Hyperspectral image classification based on domain adaptation broad learning. IEEE J Sel Top Appl Earth Obs Remote Sens 13:3006–3018

    Article  Google Scholar 

  33. Huang H et al (2022) Hyperspectral image classification via active learning and broad learning system. Appl Intell, pp 1–12

  34. Peng X, Ota K, Dong M (2020) A broad learning-driven network traffic analysis system based on fog computing paradigm. China Commun 17(2):1–13

    Article  Google Scholar 

  35. Li Q et al (2024) ScenarioNet: Open-source platform for large-scale traffic scenario simulation and modeling. Adv Neural Inf Process Syst 36

  36. Xia W et al (2022) Gan inversion: A survey. IEEE Trans Pattern Anal Mach Intell 45(3):3121–3138

    Google Scholar 

  37. Robert C (2014) Machine learning, a probabilistic perspective. CHANCE, pp 62–63

  38. Yuan H, Li G (2021) A survey of traffic prediction: from spatio-temporal data to intelligent transportation. Data Sci Eng 6:63–85

    Article  Google Scholar 

  39. Zheng G et al (2019) Learning phase competition for traffic signal control.In: Proceedings of the 28th ACM international conference on information and knowledge management

  40. Zhu R et al (2023) Multi-agent broad reinforcement learning for intelligent traffic light control. Inf Sci 619:509–525

    Article  Google Scholar 

  41. Zhu R et al (2022) Context-aware multiagent broad reinforcement learning for mixed pedestrian-vehicle adaptive traffic light control. IEEE Internet Things J 9(20):19694–19705

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China Project (62172441 and 61772553), in part by the National Natural Science Foundation of Hunan Province (2023JJ30696), in part by the local science and technology developing fundation guided by central goverment (Free exploration project 2021Szvup166), in part by the Key Project of Shenzhen City Special Fund for Fundamental Research (202208183000751), in part by the postgraduate Innovative Project of Central South University (2023XQLH003), and in part by the Opening Project of State Key Laboratory of Nickel and Cobalt Resources Comprehensive Utilization(GZSYS-KY-2022-018, GZSYS-KY-2022-024).

Author information

Author notes

  1. Shunmeng Yin and Xinjun Pei contributed equally to this work.

Authors and Affiliations

  1. School of Electronic Information, Central South University, Changsha, 410083, China

    Xiaoheng Deng, Lixin Lin, Xuechen Chen & Jinsong Gui

  2. Shenzhen Research Institute, Central South University, Shenzhen, 518000, China

    Xiaoheng Deng

  3. School of Computer Science and Engineering, Central South University, Changsha, 410083, China

    Shunmeng Yin & Xinjun Pei

Authors
  1. Xiaoheng Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shunmeng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinjun Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lixin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xuechen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jinsong Gui
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Xiaoheng Deng: Conception and design of study,Analysis and interpre-tation of Results, Writing- Original draft preparation, Review & Editing, Funding acquisition Shunmeng Yin: Acquisition of data, Coding and Implementation,Writing- Original draft preparation, Conception and design of study Xinjun Pei: Coding and Implementation, Writing- Original draft preparation, Review & Editing Lixin Lin: Conceptualization, Methodology, Analysis and interpre-tation of Results, Review & Editing, Supervision Xuechen Chen: Review & Editing, Supervision Jinsong Gui: Review & Editing, Supervision.

Corresponding author

Correspondence to Xiaoheng Deng.

Ethics declarations

Ethical and informed consent for data used

The work uses publicly available and synthetically generated datasets which do not have any identifiable information. No ethical approval was needed.

Conflicts of interest

The authors have no competing interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, X., Yin, S., Pei, X. et al. E-DBRL: efficient double broad reinforcement learning for adaptive traffic signal control. Appl Intell 54, 8563–8575 (2024). https://doi.org/10.1007/s10489-024-05637-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05637-1

Keywords

Search

Navigation