research-article

Fleet Rebalancing for Expanding Shared e-Mobility Systems: A Multi-Agent Deep Reinforcement Learning Approach

Authors:

Hongkai WenAuthors Info & Claims

IEEE Transactions on Intelligent Transportation Systems, Volume 24, Issue 4

Pages 3868 - 3881

https://doi.org/10.1109/TITS.2022.3233422

Published: 01 April 2023 Publication History

Abstract

The electrification of shared mobility has become popular across the globe. Many cities have their new shared e-mobility systems deployed, with continuously expanding coverage from central areas to the city edges. A key challenge in the operation of these systems is fleet rebalancing, i.e., how EVs should be repositioned to better satisfy future demand. This is particularly challenging in the context of expanding systems, because i) the range of the EVs is limited while charging time is typically long, which constrain the viable rebalancing operations; and ii) the EV stations in the system are dynamically changing, i.e., the legitimate targets for rebalancing operations can vary over time. We tackle these challenges by first investigating rich sets of data collected from a real-world shared e-mobility system for one year, analyzing the operation model, usage patterns and expansion dynamics of this new mobility mode. With the learned knowledge we design a high-fidelity simulator, which is able to abstract key operation details of EV sharing at fine granularity. Then we model the rebalancing task for shared e-mobility systems under continuous expansion as a Multi-Agent Reinforcement Learning (MARL) problem, which directly takes the range and charging properties of the EVs into account. We further propose a novel policy optimization approach with action cascading, which is able to cope with the expansion dynamics and solve the formulated MARL. We evaluate the proposed approach extensively, and experimental results show that our approach outperforms the state-of-the-art, offering significant performance gain in both satisfied demand and net revenue.

References

[1]

(2019). Bluecity Car Sharing. Accessed: Oct. 14, 2019. [Online]. Available: https://www.blue-city.co.uk/

[2]

(2019). Volkswagen Starts ‘We Share’ E-Mobility Car Sharing in Berlin. Accessed: Oct. 14, 2019. [Online]. Available: https://www.volkswagenag.com/en/news/2018/08/VW_Brand_We_Share.html

[3]

(2019). Bluesg. Accessed: Oct. 14, 2019. [Online]. Available: https://www.bluesg.com.sg/

[4]

S. Ghosh, P. Varakantham, Y. Adulyasak, and P. Jaillet, “Dynamic repositioning to reduce lost demand in bike sharing systems,” J. Artif. Intell. Res., vol. 58, pp. 387–430, Feb. 2017.

[5]

Y. Li, Y. Zheng, and Q. Yang, “Dynamic bike reposition: A spatio-temporal reinforcement learning approach,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 1724–1733.

[6]

S. Ghosh, J. Y. Koh, and P. Jaillet, “Improving customer satisfaction in bike sharing systems through dynamic repositioning,” in Proc. 28th Int. Joint Conf. Artif. Intell. (IJCAI). International Joint Conferences on Artificial Intelligence Organization, Jul. 2019, pp. 5864–5870. 10.24963/ijcai.2019/813.

[7]

K. Lin, R. Zhao, Z. Xu, and J. Zhou, “Efficient large-scale fleet management via multi-agent deep reinforcement learning,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 1774–1783.

[8]

M. Liet al., “Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning,” in Proc. World Wide Web Conf., May 2019, pp. 983–994.

[9]

C. Wei, Y. Wang, X. Yan, and C. Shao, “Look-ahead insertion policy for a shared-taxi system based on reinforcement learning,” IEEE Access, vol. 6, pp. 5716–5726, 2018.

[10]

F. Kooti, M. Grbovic, L. M. Aiello, N. Djuric, V. Radosavljevic, and K. Lerman, “Analyzing Uber’s ride-sharing economy,” in Proc. 26th Int. Conf. World Wide Web Companion (WWW Companion), 2017, pp. 574–582.

[11]

S. Jiang, L. Chen, A. Mislove, and C. Wilson, “On ridesharing competition and accessibility: Evidence from Uber, Lyft, and taxi,” in Proc. World Wide Web Conf. World Wide Web (WWW), 2018, pp. 863–872.

[12]

A. Singla, M. Santoni, G. Bartók, P. Mukerji, M. Meenen, and A. Krause, “Incentivizing users for balancing bike sharing systems,” in Proc. 29th AAAI Conf. Artif. Intell., 2015, pp. 1–7.

[13]

L. Pan, Q. Cai, Z. Fang, P. Tang, and L. Huang, “A deep reinforcement learning framework for rebalancing dockless bike sharing systems,” in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019, pp. 1393–1400.

[14]

O. Tremblay and L.-A. Dessaint, “Experimental validation of a battery dynamic model for EV applications,” World Electr. Vehicle J., vol. 3, pp. 289–298, May 2009.

[15]

S. Albrecht and P. Stone, “Multiagent learning: Foundations and recent trends,” in Proc. Tutorial IJCAI Conf., vol. 223, 2017, pp. 1–186.

[16]

V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” in Proc. Adv. Neural Inf. Process. Syst., 2000, pp. 1008–1014.

[17]

D. R. Hunter and K. Lange, “A tutorial on MM algorithms,” Amer. Statist., vol. 58, no. 1, pp. 30–37, 2004.

[18]

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017, arXiv:1707.06347.

[19]

M. Luo, B. Du, K. Klemmer, H. Zhu, H. Ferhatosmanoglu, and H. Wen, “D3P: Data-driven demand prediction for fast expanding electric vehicle sharing systems,” Proc. ACM Interact., Mobile, Wearable Ubiquitous Technol., vol. 4, no. 1, pp. 1–21, Mar. 2020.

Digital Library

[20]

E. Lianget al., “Rllib: Abstractions for distributed reinforcement learning,” in Proc. Int. Conf. Mach. Learn., 2018, pp. 3053–3062.

[21]

Z. Xuet al., “Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 905–913.

[22]

M. Furuhata, M. Dessouky, F. Ordóñez, M.-E. Brunet, X. Wang, and S. Koenig, “Ridesharing: The state-of-the-art and future directions,” Transp. Res. B, Methodol., vol. 57, pp. 28–46, Nov. 2013.

[23]

Z. Li, Y. Hong, and Z. Zhang, “An empirical analysis of on-demand ride sharing and traffic congestion,” in Proc. Int. Conf. Inf. Syst., 2016, pp. 1–10.

[24]

T. R. Dillahunt, X. Wang, E. Wheeler, H. F. Cheng, B. Hecht, and H. Zhu, “The sharing economy in computing: A systematic literature review,” Proc. ACM Hum.-Comput. Interact., vol. 1, p. 38, Dec. 2017.

[25]

A. Sarker, H. Shen, and J. A. Stankovic, “MORP: Data-driven multi-objective route planning and optimization for electric vehicles,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 1, no. 4, p. 162, Jan. 2018. 10.1145/3161408.

Digital Library

[26]

C. F. Yuen, A. P. Singh, S. Goyal, S. Ranu, and A. Bagchi, “Beyond shortest paths: Route recommendations for ride-sharing,” in Proc. World Wide Web Conf. (WWW), 2019, pp. 2258–2269.

[27]

L. Yanet al., “Employing opportunistic charging for electric taxicabs to reduce idle time,” Proc. ACM Interact. Mobile Wearable Ubiquitous Technol., vol. 2, no. 1, p. 47, 2018.

[28]

Y. Yuan, D. Zhang, F. Miao, J. Chen, T. He, and S. Lin, “P²Charging: Proactive partial charging for electric taxi systems,” in Proc. IEEE 39th Int. Conf. Distrib. Comput. Syst. (ICDCS), Jul. 2019, pp. 1–13.

[29]

G. Wanget al., “SharedCharging: Data-driven shared charging for large-scale heterogeneous electric vehicle fleets,” Proc. ACM Interact., Mobile, Wearable Ubiquitous Technol., vol. 3, no. 3, pp. 1–25, Sep. 2019.

Digital Library

[30]

B. Du, Y. Tong, Z. Zhou, Q. Tao, and W. Zhou, “Demand-aware charger planning for electric vehicle sharing,” in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Jul. 2018, pp. 1330–1338.

[31]

G. Wang, X. Chen, F. Zhang, Y. Wang, and D. Zhang, “Experience: Understanding long-term evolving patterns of shared electric vehicle networks,” in Proc. 25th Annu. Int. Conf. Mobile Comput. Netw., Aug. 2019, pp. 1–12.

[32]

J. Liu, L. Sun, W. Chen, and H. Xiong, “Rebalancing bike sharing systems: A multi-source data smart optimization,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2016, pp. 1005–1014.

[33]

T. Raviv, M. Tzur, and I. A. Forma, “Static repositioning in a bike-sharing system: Models and solution approaches,” EURO J. Transp. Logistics, vol. 2, no. 3, pp. 187–229, Aug. 2013.

[34]

L. Zhanget al., “A taxi order dispatch model based on combinatorial optimization,” in Proc. 23rd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Aug. 2017, pp. 2151–2159.

[35]

S. Ghosh, M. Trick, and P. Varakantham, “Robust repositioning to counter unpredictable demand in bike sharing systems,” in Proc. 25th Int. Joint Conf. Artif. Intell. (IJCAI). New York, NY, USA: AAAI Press, 2016, pp. 3096–3102.

[36]

S. Wang, T. He, D. Zhang, Y. Liu, and S. H. Son, “Towards efficient sharing: A usage balancing mechanism for bike sharing systems,” in Proc. World Wide Web Conf., May 2019, pp. 2011–2021.

[37]

C. Etienne and O. Latifa, “Model-based count series clustering for bike sharing system usage mining: A case study with the vélib’system of Paris,” ACM Trans. Intell. Syst. Technol., vol. 5, no. 3, p. 39, 2014.

[38]

D. Chemla, F. Meunier, T. Pradeau, R. W. Calvo, and H. Yahiaoui, “Self-service bike sharing systems: Simulation, repositioning, pricing,” Working Paper, Mar. 2013. [Online]. Available: https://hal.archives-ouvertes.fr/hal-00824078

[39]

B. Bakker, S. Whiteson, L. Kester, and F. C. A. Groen, “Traffic light control by multiagent reinforcement learning systems,” in Interactive Collaborative Information Systems, R. Babuška and F. C. A. Groen, Eds. Berlin, Germany: Springer, 2010, pp. 475–510. 10.1007/978-3-642-11688-9_18.

Cited By

Sun KGao XLi J(2024)The Analysis of Smarter FutureJournal of Organizational and End User Computing10.4018/JOEUC.35724936:1(1-20)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.4018/JOEUC.357249
Song C(2024)The Role of Transformer-cGANs Fusion in Digital MarketingJournal of Organizational and End User Computing10.4018/JOEUC.34791436:1(1-22)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.4018/JOEUC.347914
Feng X(2024)Design and Application of Virtual Cloud OMS Computing in Smart AirportInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.34766617:1(1-15)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.4018/IJITSA.347666
Show More Cited By

Recommendations

Rebalancing expanding EV sharing systems with deep reinforcement learning
IJCAI'20: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence

Electric Vehicle (EV) sharing systems have recently experienced unprecedented growth across the world. One of the key challenges in their operation is vehicle rebalancing, i.e., repositioning the EVs across stations to better satisfy future user demand. ...
Multi-Agent Reinforcement Learning for Intelligent V2G Integration in Future Transportation Systems
Electric vehicles (EVs) are the backbone of the future intelligent transportation system (ITS). They are environmentally friendly and can also be integrated as distributed energy resources (DERs) into the smart grid using vehicle-to-grid (V2G) scheme. ...
Distributed Optimization of Solar Micro-grid Using Multi Agent Reinforcement Learning
Abstract
In the distributed optimization of micro-grid, we consider grid connected solar micro-grid system which contains a local consumer, a solar photovoltaic system and a battery. The consumer as an agent continuously interacts with the environment and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Intelligent Transportation Systems

IEEE Transactions on Intelligent Transportation Systems Volume 24, Issue 4

April 2023

1080 pages

ISSN:1524-9050

Issue’s Table of Contents

1558-0016 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 April 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun KGao XLi J(2024)The Analysis of Smarter FutureJournal of Organizational and End User Computing10.4018/JOEUC.35724936:1(1-20)Online publication date: 7-Nov-2024
https://dl.acm.org/doi/10.4018/JOEUC.357249
Song C(2024)The Role of Transformer-cGANs Fusion in Digital MarketingJournal of Organizational and End User Computing10.4018/JOEUC.34791436:1(1-22)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.4018/JOEUC.347914
Feng X(2024)Design and Application of Virtual Cloud OMS Computing in Smart AirportInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.34766617:1(1-15)Online publication date: 17-Sep-2024
https://dl.acm.org/doi/10.4018/IJITSA.347666
Guo KWu MLi XSong HKumar N(2023)Deep Reinforcement Learning and NOMA-Based Multi-Objective RIS-Assisted IS-UAV-TNs: Trajectory Optimization and Beamforming DesignIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.326760724:9(10197-10210)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TITS.2023.3267607

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents