Abstract
This article studies the pump scheduling optimization problem in water distribution networks (WDNs) through a novel algorithm that combines knowledge learning and deep reinforcement learning. The optimization problem is modeled as a Markov decision process by taking three objectives in pressure management into consideration. Knowledge-assisted learning is incorporated into the reinforcement learning framework (KA-RL) to help evaluate the state value and guide the design of reward function, since the proposed KA-RL framework leverages the notion that historical data of WDNs could be utilized to produce optimal trajectories with respect to parametric variations. The knowledge-assisted proximal policy optimization (KA-PPO) algorithm, which only uses nodal pressure data, is proposed based on the KA-RL framework to address the arbitrary WDN topology and time-varying water demand. The effectiveness and applicability of the proposed algorithm are illustrated by virtue of the 22-node network with two pumps in a pump station. Empirical results demonstrate that KA-PPO works well in practice and compares favorably to Nelder–Mead method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Abdallah M, Kapelan Z (2019) Fast pump scheduling method for optimum energy cost and water quality in water distribution networks with fixed and variable speed pumps. J Water Resour Plann Manag 145(12):04019055
Barán B, Von Lücken C, Sotelo A (2005) Multi-objective pump scheduling optimisation using evolutionary strategies. Adv Eng Softw 36(1):39–47
Bazrafshan M, Gatsis N, Giacomoni M, Taha A (2018) A fixed-point iteration for steady-state analysis of water distribution networks. In: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), IEEE, pp 880–884
Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38(8):716
Bene J, Selek I, Hős C (2013) Comparison of deterministic and heuristic optimization solvers for water network scheduling problems. Water Sci Technol Water Supply 13(5):1367–1376
Broad D, Maier HR, Dandy GC (2010) Optimal operation of complex water distribution systems using metamodels. J Water Resour Plann Manag 136(4):433–443
Guo G, Liu S, Wu Y, Li J, Zhou R, Zhu X (2018) Short-term water demand forecast based on deep learning method. J Water Resour Plann Manag 144(12):04018076
Hajgató G, Paál G, Gyires-Tóth B (2020) Deep reinforcement learning for real-time optimization of pumps in water distribution systems. J Water Resour Plann Manag 146(11):04020079
Hashemi SS, Tabesh M, Ataeekia B (2014) Ant-colony optimization of pumping schedule to minimize the energy cost using variable-speed pumps in water distribution networks. Urban Water J 11(5):335–347
Kougias IP, Theodossiou NP (2013) Multiobjective pump scheduling optimization using harmony search algorithm (hsa) and polyphonic hsa. Water Resour Manag 27(5):1249–1261
Levine S, Finn C, Darrell T, Abbeel P (2016) End-to-end training of deep visuomotor policies. The J Mach Learn Res 17(1):1334–1373
Liu D, Li S, Zheng Y, Wu J (2019) Distributed zone mpc of pressure management for water distribution network systems. IET Control Theory Appl 13(11):1704–1717
Liu D, Zheng Y, Wu J, Li S (2020) Zone model predictive control for pressure management of water distribution network. Asian J Control 22(4):1522–1536
Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26
Mala-Jetmarova H, Sultanova N, Savic D (2017) Lost in optimisation of water distribution systems? A literature review of system operation. Environ Modell Softw 93:209–254
Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533
Odan FK, Ribeiro Reis LF, Kapelan Z (2015) Real-time multiobjective optimization of operation of water supply systems. J Water Resour Plann Manag 141(9):04015011
Reddy T, RM SP, Parimala M, Chowdhary CL, Hakak S, Khan WZ, et al (2020) A deep neural networks based model for uninterrupted marine environment monitoring. Comput Commun 157:64–75
Ritter J, Corzo G, Solomatine DP, Angarita H (2020) Multiobjective direct policy search using physically based operating rules in multireservoir systems. J Water Resour Plann Manag 146(4):05020002
Rossman L (1993) Epanet–an advanced water quality modeling package for distribution systems. In: Proceeding management and regulations, american water works annual conference, pp 41–418
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms
Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489
Singh MK, Kekatos V (2020) On the flow problem in water distribution networks: uniqueness and solvers. IEEE Trans Control Netw Syst
Stokes CS, Maier HR, Simpson AR (2015) Water distribution system pumping operational greenhouse gas emissions minimization by considering time-dependent emissions factors. J Water Resour Plann Manag 141(7):04014088
Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT press, Cambridge
Walski TM, Brill ED Jr, Gessler J, Goulter IC, Jeppson RM, Lansey K, Lee HL, Liebman JC, Mays L, Morgan DR et al (1987) Battle of the network models: epilogue. J Water Resour Plan Manag 113(2):191–203
Wang S, Taha A, Gatsis N, Giacomoni M (2020) Receding horizon control for drinking water networks: the case for geometric programming. IEEE Trans Control Netw Syst
Wang Y, Ocampo-Martinez C, Puig V (2016) Stochastic model predictive control based on gaussian processes applied to drinking water networks. IET Control Theory Appl 10(8):947–955
Wang Y, Puig V, Cembrano G (2017) Non-linear economic model predictive control of water distribution networks. J Process Control 56:23–34
Ye D, Chen G, Zhang W, Chen S, Yuan B, Liu B, Chen J, Liu Z, Qiu F, Yu H, et al. (2020) Towards playing full moba games with deep reinforcement learning. Adv Neural Inform Process Syst 33
Zhao H, Zhao J, Qiu J, Liang G, Dong ZY (2020) Cooperative wind farm control with deep reinforcement learning and knowledge assisted learning. IEEE Trans Ind Inform
Zhu Y, Mottaghi R, Kolve E, Lim JJ, Gupta A, Fei-Fei L, Farhadi A (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 3357–3364
Acknowledgements
This work is supported by National Natural Science Foundation of China (61633019).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interests regarding the publication of this paper.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Data Availability
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xu, J., Wang, H., Rao, J. et al. Zone scheduling optimization of pumps in water distribution networks with deep reinforcement learning and knowledge-assisted learning. Soft Comput 25, 14757–14767 (2021). https://doi.org/10.1007/s00500-021-06177-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-021-06177-3