Abstract
The economic profitability of Smart Grid prosumers (i.e., producers that are simultaneously consumers) depends on their tackling of the decision-making problem they face when selling and buying energy. In previous work, we had modelled this problem compactly as a factored Markov Decision Process, capturing the main aspects of the business decisions of a prosumer corresponding to a community microgrid of any size. Though that work had employed an exact value iteration algorithm to obtain a near-optimal solution over discrete state spaces, it could not tackle problems defined over continuous state spaces. By contrast, in this paper we show how to use approximate MDP solution methods for taking decisions in this domain without the need of discretizing the state space. Specifically, we employ fitted value iteration, a sampling-based approximation method that is known to be well behaved. By so doing, we generalize our factored MDP solution method to continuous state spaces. We evaluate our approach using a variety of basis functions over different state sample sizes, and compare its performance to that of our original “exact” value iteration algorithm. Our generic approximation method is shown to exhibit stable performance in terms of accumulated reward, which for certain basis functions reaches 98 % of that gathered by the exact algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
See http://www.powertac.org/node/11 for a list of related publications.
- 2.
States on the x axis in these figures are ranked in reverse order wrt. steps-to-go in the horizon: states with small indices occur early in the day-ahead, and the ones to the right late.
References
Ackermann, T. (ed.): Wind Power in Power Systems. Wiley, Chichester (2005)
Angelidakis, A., Chalkiadakis, G.: Factored MDPs for optimal prosumer decision-making. In: Proceedings of AAMAS-2015, pp. 503–511 (2015)
Asmus, P.: Microgrids, virtual power plants and our distributed energy future. Electr. J. 23(10), 72–82 (2010)
Boutilier, C., Dean, T., Hanks, S.: Decision-theoretic planning: structural assumptions and computational leverage. J. Artif. Intell. Res. (JAIR) 11, 1–94 (1999)
Busoniu, L., Babuska, R., De Schutter, B., Ernst, D.: Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Boca Raton (2010)
Chalkiadakis, G., Robu, V., Kota, R., Rogers, A., Jennings, N.: Cooperatives of distributed energy resources for efficient virtual power plants. In: Proceedings of AAMAS-2011, pp. 787–794 (2011)
DeGroot, M., Schervish, J.: Probability and Statistics. Addison-Wesley, Reading (2002)
Gordon, G.J.: Stable function approximation in dynamic programming. In: Proceedings of the 12th International Conference on Machine Learning, pp. 261–268 (1995)
Guestrin, C., Koller, D., Parr, R., Venkataraman, S.: Efficient solution algorithms for factored MDPs. J. Artif. Intell. Res. (JAIR) 19, 399–468 (2003)
Kanchev, H., Lu, D., Colas, F., Lazarov, V., Francois, B.: Energy management and operational planning of a microgrid with a PV-based active generator for Smart Grid applications. IEEE Trans. Ind. Electron. 58(10), 4583–4592 (2011)
Kirschen, D., Strbac, G.: Fundamentals of Power System Economics. Wiley, Chichester (2005)
Munos, R., Szepesvári, C.: Finite-time bounds for fitted value iteration. J. Mach. Learn. Res. 9, 815–857 (2008)
Nikovski, D., Zhang, W.: Factored markov decision process models for stochastic unit commitment. In: IEEE Conference on Innovative Technologies for an Efficient and Reliable Electricity Supply (CITRES), pp. 28–35 (2010)
Ramchurn, S.D., Vytelingum, P., Rogers, A., Jennings, N.: Putting the ‘smarts’ into the Smart Grid: a grand challenge for artificial intelligence. Commun. ACM 55(4), 86–97 (2012)
Rogers, A., Ramchurn, S., Jennings, N.: Delivering the Smart Grid: challenges for autonomous agents and multi-agent systems research. In: Proceedings of AAAI-2012, pp. 2166–2172 (2012)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)
Zhao, B., Zhang, X., Chen, J., Wang, C., Guo, L.: Operation optimization of standalonemicrogrids considering lifetime characteristics of battery energy storage system. IEEE Trans.Sustain. Energ. 4(4), 934–943 (2013)
Federation of European renewable energy cooperatives. http://www.rescoop.eu
Acknowledgements
The work presented in this paper was supported by the Greek General Secretariat for Research and Technology (GSRT) through the funding of research project “AFORMI – Reconfigurable Systems for scientific research” with proposal code 2427 within the context of action “ARISTEIA” of the Lifelong Learning Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Angelidakis, A., Chalkiadakis, G. (2016). Factored MDPs for Optimal Prosumer Decision-Making in Continuous State Spaces. In: Rovatsos, M., Vouros, G., Julian, V. (eds) Multi-Agent Systems and Agreement Technologies. EUMAS AT 2015 2015. Lecture Notes in Computer Science(), vol 9571. Springer, Cham. https://doi.org/10.1007/978-3-319-33509-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-33509-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-33508-7
Online ISBN: 978-3-319-33509-4
eBook Packages: Computer ScienceComputer Science (R0)