research-article

Free access

Mitigating Action Hysteresis in Traffic Signal Control with Traffic Predictive Reinforcement Learning

Authors:

Wanyu WangAuthors Info & Claims

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 673 - 684

https://doi.org/10.1145/3580305.3599528

Published: 04 August 2023 Publication History

Abstract

Traffic signal control plays a pivotal role in the management of urban traffic flow. With the rapid advancement of reinforcement learning, the development of signal control methods has seen a significant boost. However, a major challenge in implementing these methods is ensuring that signal lights do not change abruptly, as this can lead to traffic accidents. To mitigate this risk, a time-delay is introduced in the implementation of control actions, but usually has a negative impact on the overall efficacy of the control policy. To address this challenge, this paper presents a novel Traffic Signal Control Framework (PRLight), which leverages an On-policy Traffic Control Model (OTCM) and an Online Traffic Prediction Model (OTPM) to achieve efficient and real-time control of traffic signals. The framework collects multi-source traffic information from a local-view graph in real-time and employs a novel fast attention mechanism to extract relevant traffic features. To be specific, OTCM utilizes the predicted traffic state as input, eliminating the need for communication with other agents and maximizing computational efficiency while ensuring that the most relevant information is used for signal control. The proposed framework was evaluated on both simulated and real-world road networks and compared to various state-of-the-art methods, demonstrating its effectiveness in preventing traffic congestion and accidents.

Supplementary Material

MP4 File (758-2min-promo.mp4)

Presentation video - short version

Download
37.78 MB

References

[1]

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, Vol. 34, 6 (2017), 26--38.

[2]

Andrew G Barto, Richard S Sutton, and Charles W Anderson. 1983. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics 5 (1983), 834--846.

[3]

Yu-Chiun Chiou and Chien-Hua Chang. 2010. Driver responses to green and red vehicular signal countdown displays: Safety and efficiency aspects. Accident Analysis & Prevention, Vol. 42, 4 (2010), 1057--1065.

[4]

Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. 2019. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, Vol. 21, 3 (2019), 1086--1095.

[5]

Seung-Bae Cools, Carlos Gershenson, and Bart D'Hooghe. 2013. Self-organizing traffic lights: A realistic simulation. In Advances in applied self-organizing systems. Springer, 45--55.

[6]

Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. 2020. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 (2020).

[7]

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.

[8]

David L Gerlough and Matthew J Huber. 1976. Traffic flow theory. Technical Report.

[9]

Qiangqiang Guo, Li Li, and Xuegang (Jeff) Ban. 2019. Urban traffic signal control with connected and automated vehicles: A survey. Transportation Research Part C: Emerging Technologies, Vol. 101 (2019), 313--334. https://doi.org/10.1016/j.trc.2019.01.026

[10]

Ke Han and Vikash V Gayah. 2015. Continuum signalized junction model for dynamic traffic networks: Offset, spillback, and multiple signal phases. Transportation Research Part B: Methodological, Vol. 77 (2015), 213--239.

[11]

Xiao Han, Guojiang Shen, Xi Yang, and Xiangjie Kong. 2020. Congestion recognition for hybrid urban road systems via digraph convolutional network. Transportation Research Part C: Emerging Technologies, Vol. 121 (2020), 102877.

[12]

JR Head. 1982. A new specification for inductive loop detectors. Traffic Engineering & Control, Vol. 23, 4 (1982).

[13]

Jingzhi Hu, Hongliang Zhang, Lingyang Song, Zhu Han, and H Vincent Poor. 2020a. Reinforcement learning for a cellular internet of UAVs: Protocol design, trajectory control, and resource management. IEEE Wireless Communications, Vol. 27, 1 (2020), 116--123.

Digital Library

[14]

Jingzhi Hu, Hongliang Zhang, Lingyang Song, Robert Schober, and H Vincent Poor. 2020b. Cooperative internet of UAVs: Distributed trajectory design by multi-agent deep reinforcement learning. IEEE Transactions on Communications, Vol. 68, 11 (2020), 6807--6821.

[15]

Qize Jiang, Minhao Qin, Shengmin Shi, Weiwei Sun, and Baihua Zheng. 2022. Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, Lud De Raedt (Ed.). 3854--3860.

[16]

Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, Vol. 4 (1996), 237--285.

Digital Library

[17]

Mohamed A Khamis, Walid Gomaa, and Hisham El-Shishiny. 2012. Multi-objective traffic light control system based on Bayesian probability interpretation. In 2012 15th International IEEE conference on intelligent transportation systems. IEEE, 995--1000.

[18]

Neetesh Kumar, Syed Shameerur Rahman, and Navin Dhakad. 2020. Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system. IEEE Transactions on Intelligent Transportation Systems, Vol. 22, 8 (2020), 4919--4928.

Digital Library

[19]

Xiaoyuan Liang, Xunsheng Du, Guiling Wang, and Zhu Han. 2019. A deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, Vol. 68, 2 (2019), 1243--1253.

[20]

Alan J. Miller. [n.,d.]. Settings for Fixed-Cycle Traffic Signals., Vol. 14, 4 ([n.,d.]), 373--386. https://doi.org/10.1057/jors.1963.61

[21]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).

[22]

Tomoki Nishi, Keisuke Otaki, Keiichiro Hayakawa, and Takayoshi Yoshimura. 2018. Traffic signal control based on reinforcement learning with graph convolutional neural nets. In 2018 21st International conference on intelligent transportation systems (ITSC). IEEE, 877--883.

Digital Library

[23]

Isaac Porche and Stéphane Lafortune. 1999. Adaptive look-ahead optimization of traffic signals. Journal of Intelligent Transportation System, Vol. 4, 3--4 (1999), 209--254.

[24]

As' ad Salkham and Vinny Cahill. 2010. Soilse: A decentralized approach to optimization of fluctuating urban traffic using reinforcement learning. In 13th international IEEE conference on intelligent transportation systems. IEEE, 531--538.

[25]

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015).

[26]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[27]

Guojiang Shen, Xiao Han, KwaiSang Chin, and Xiangjie Kong. 2021. An attention-based digraph convolution network enabled framework for congestion recognition in three-dimensional road networks. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, 9 (2021), 14413--14426.

Digital Library

[28]

Yifan Tang and Yan Xu. 2021. Multi-agent deep reinforcement learning for solving large-scale air traffic flow management problem: A time-step sequential decision approach. In 2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC). IEEE, 1--10.

[29]

Anna Izabel J Tostes, Fátima de LP Duarte-Figueiredo, Renato Assuncc ao, Juliana Salles, and Antonio AF Loureiro. 2013. From data to knowledge: City-wide traffic flows analysis and prediction using bing maps. In Proceedings of the 2nd ACM SIGKDD international workshop on urban computing. 1--8.

Digital Library

[30]

Elise Van der Pol and Frans A Oliehoek. 2016. Coordinated deep reinforcement learners for traffic light control. Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016), Vol. 1 (2016).

[31]

Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30.

[32]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).

[33]

Fo Vo Webster. 1958. Traffic signal settings. Technical Report.

[34]

Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. 2019. Colight: Learning network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1913--1922.

Digital Library

[35]

Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. 2018. Intellilight: A reinforcement learning approach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2496--2505.

Digital Library

[36]

Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, Vol. 8, 3 (1992), 229--256.

Digital Library

[37]

Cathy Wu, Kanaad Parvate, Nishant Kheterpal, Leah Dickstein, Ankur Mehta, Eugene Vinitsky, and Alexandre M Bayen. 2017. Framework for control and deep reinforcement learning in traffic. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 1--8.

Digital Library

[38]

Libing Wu, Min Wang, Dan Wu, and Jia Wu. 2021. DynSTGAT: Dynamic Spatial-Temporal Graph Attention Network for Traffic Signal Control. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2150--2159.

Digital Library

[39]

Richard Wunderlich, Cuibi Liu, Itamar Elhanany, and Tom Urbanik. 2008. A novel signal-scheduling algorithm with quality-of-service provisioning for an isolated intersection. IEEE Transactions on Intelligent Transportation Systems, Vol. 9, 3 (2008), 536--547.

Digital Library

[40]

Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu. 2021. Hierarchically and cooperatively learning traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 669--677.

[41]

Y Yamazaki, K Shigematsu, S Kato, F Kojima, H Onari, and S Takata. 2017. Design method of material handling systems for lean automation-Integrating equipment for reducing wasted waiting time. CIRP Annals, Vol. 66, 1 (2017), 449--452.

[42]

Kok-Lim Alvin Yau, Junaid Qadir, Hooi Ling Khoo, Mee Hong Ling, and Peter Komisarczuk. 2017. A survey on reinforcement learning models and algorithms for traffic signal control. ACM Computing Surveys (CSUR), Vol. 50, 3 (2017), 1--38.

[43]

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, Vol. 35 (2022), 24611--24624.

[44]

Zhengxu Yu, Shuxian Liang, Long Wei, Zhongming Jin, Jianqiang Huang, Deng Cai, Xiaofei He, and Xian-Sheng Hua. 2021. Macar: Urban traffic light control via active multi-agent communication and action rectification. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 2491--2497.

[45]

Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, and Zhenhui Li. 2020. Metalight: Value-based meta-reinforcement learning for traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 1153--1160.

[46]

Zijian Zhang, Xiangyu Zhao, Hao Miao, Chunxu Zhang, Hongwei Zhao, and Junbo Zhang. 2023. AutoSTL: Automated Spatio-Temporal Multi-Task Learning. In Proceedings of the AAAI Conference on Artificial Intelligence.

Digital Library

[47]

Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. 2019. Learning phase competition for traffic signal control. In Proceedings of the 28th ACM international conference on information and knowledge management. 1963--1972.

Digital Library

Cited By

Zhu YYu JZhao XLiu QYe YChen WZhang ZWei XLiang YBaeza-Yates RBonchi F(2024)ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion ModelProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671866(4676-4687)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671866
Zhu YYe YZhang SZhao XYu JOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)DiffTrajProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668965(65168-65188)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668965
Zhang ZZhao XLiu QZhang CMa QWang WZhao HWang YLiu ZFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)PromptST: Prompt-Enhanced Spatio-Temporal Multi-Attribute PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615016(3195-3205)Online publication date: 21-Oct-2023
https://doi.org/10.1145/3583780.3615016
Show More Cited By

Index Terms

Mitigating Action Hysteresis in Traffic Signal Control with Traffic Predictive Reinforcement Learning
1. Applied computing
  1. Operations research
    1. Transportation
2. Information systems
  1. Information systems applications
    1. Spatial-temporal systems

Recommendations

Hierarchical traffic signal optimization using reinforcement learning and traffic prediction with long-short term memory
Abstract
Multi-agent systems can be used for modelling large-scale distributed systems in real world applications. In intelligent transportation system (ITS), many interacting entities influence the performance of the system. As part of ITS, ...
A DQN Algorithm for Traffic Signal Control Based on Meta-learning Training
ICDSP '23: Proceedings of the 2023 7th International Conference on Digital Signal Processing

With the rapid development of our society and economy, traffic congestion caused by the increase in vehicles is an important challenge we are facing today. Traffic control, as an integral part of intelligent transport systems (ITS), plays a huge role in ...
Reinforcement learning for traffic signal control: Incorporating a virtual mesoscopic model for depicting oversaturated traffic conditions
Abstract
Recently, with increasing urban traffic congestion, there has been an upsurge in studies on reinforcement learning for traffic signal control (RL-TSC), which enables efficient traffic management. However, most existing RL-TSC research relies on ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2023

5996 pages

ISBN:9798400701030

DOI:10.1145/3580305

General Chairs:
Ambuj Singh
UC Santa Barbara, USA
,
Yizhou Sun
UC Los Angeles, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Dimitrios Gunopulos
University of Athens, Greece
,
Xifeng Yan
UC Santa Barbara, USA
,
Ravi Kumar
Google, USA
,
Fatma Ozcan
Google, USA
,
Jieping Ye
Alibaba DAMO Academy

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

SIRG - CityU Strategic Interdisciplinary Research Grant
CityU - HKIDS Early Career Research Grant
Ant Group (CCF-Ant Research Fund, Ant Group Research Fund)
CCF-Tencent Open Fund
APRC - CityU New Research Initiatives
Huawei (Huawei Innovation Research Program)

Conference

KDD '23

Sponsor:

KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 6 - 10, 2023

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
807
Total Downloads

Downloads (Last 12 months)486
Downloads (Last 6 weeks)43

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhu YYu JZhao XLiu QYe YChen WZhang ZWei XLiang YBaeza-Yates RBonchi F(2024)ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion ModelProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671866(4676-4687)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671866
Zhu YYe YZhang SZhao XYu JOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)DiffTrajProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668965(65168-65188)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668965
Zhang ZZhao XLiu QZhang CMa QWang WZhao HWang YLiu ZFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)PromptST: Prompt-Enhanced Spatio-Temporal Multi-Attribute PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615016(3195-3205)Online publication date: 21-Oct-2023
https://doi.org/10.1145/3583780.3615016
Zhang ZHuang ZHu ZZhao XWang WLiu ZZhang JQin SZhao HFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)MLPST: MLP is All You Need for Spatio-Temporal PredictionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614969(3381-3390)Online publication date: 21-Oct-2023
https://doi.org/10.1145/3583780.3614969

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents