short-paper

Open access

Pricing in Prosumer Aggregations using Reinforcement Learning

Authors:

Utkarsha Agwan,

Lucas Spangher,

William Arnold,

Tarang Srivastava,

Kameshwar Poolla,

Costas J. SpanosAuthors Info & Claims

e-Energy '21: Proceedings of the Twelfth ACM International Conference on Future Energy Systems

Pages 220 - 224

https://doi.org/10.1145/3447555.3464853

Published: 22 June 2021 Publication History

Abstract

Prosumers with generation and storage capabilities can supply energy back to the grid, or trade their surplus with other prosumers for their mutual benefit. A prosumer aggregation that facilitates such trades will price the energy being traded to achieve an objective such as profit maximization, social welfare, or market equilibrium. We propose the use of reinforcement learning to design a transactive controller to price energy in a prosumer aggregation. This has an advantage over other decentralized pricing mechanisms as it does not rely on iterative price settlement or load estimation by prosumers, and estimates the price in a day ahead manner. We present numerical case studies to evaluate our controller, and discuss extensions to implement this in real prosumer aggregations.

References

[1]

Utkarsha Agwan. 2020. Optimal Prosumer Aggregations: Design and Modeling. (2020).

[2]

Edoardo Bacci and David Parker. 2020. Probabilistic guarantees for safe deep reinforcement learning. In International Conference on Formal Modeling and Analysis of Timed Systems. Springer, 231--248.

Digital Library

[3]

Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. 2019. Policy certificates: Towards accountable reinforcement learning. In International Conference on Machine Learning. PMLR, 1507--1516.

[4]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. CoRR abs/1703.03400 (2017). arXiv:1703.03400 http://arxiv.org/abs/1703.03400

[5]

Andrew Forney and Elias Bareinboim. 2019. Counterfactual Randomization: Rescuing Experimental Studies from Obscured Confounding. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 2454--2461. https://doi.org/10.1609/aaai.v33i01.33012454

Digital Library

[6]

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. PMLR, 1861--1870.

[7]

Rodrigo Henriquez-Auba, Patricia Pauli, Dileep Kalathil, Duncan S Callaway, and Kameshwar Poolla. 2018. The Sharing Economy for Residential Solar Generation. In 2018 IEEE Conference on Decision and Control (CDC). IEEE, 7322--7329.

[8]

Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. 2018. Stable Baselines. https://github.com/hill-a/stable-baselines.

[9]

Liyan Jia, Qing Zhao, and Lang Tong. 2013. Retail pricing for stochastic demand with unknown parameters: An online machine learning approach. In 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 1353--1358.

[10]

Seung-Jun Kim and Geogios B Giannakis. 2016. An online convex optimization approach to real-time energy pricing for demand response. IEEE Transactions on Smart Grid 8, 6 (2016), 2784--2793.

[11]

P. Kofinas, A.I. Dounis, and G.A. Vouros. 2018. Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids. Applied Energy 219 (2018), 53--67. https://doi.org/10.1016/j.apenergy.2018.03.017

[12]

Vijay R Konda and John N Tsitsiklis. 2000. Actor-critic algorithms. In Advances in neural information processing systems. Citeseer, 1008--1014.

[13]

Nian Liu, Xinghuo Yu, Cheng Wang, Chaojie Li, Li Ma, and Jinyong Lei. 2017. Energy-sharing model with price-based demand response for microgrids of peer-to-peer prosumers. IEEE Transactions on Power Systems 32, 5 (2017), 3569--3583.

[14]

Renzhi Lu, Seung Ho Hong, and Xiongfeng Zhang. 2018. A dynamic pricing demand response algorithm for smart grid: reinforcement learning approach. Applied Energy 220 (2018), 220--230.

[15]

Brida V. Mbuwir, Frederik Ruelens, Fred Spiessens, and Geert Deconinck. 2017. Battery Energy Management in a Microgrid Using Batch Reinforcement Learning. Energies 10, 11 (2017). https://www.mdpi.com/1996-1073/10/11/1846

[16]

Clayton Miller and Forrest Meggers. 2017. The Building Data Genome Project: An open, public data set from non-residential building electrical meters. Energy Procedia 122 (2017), 439--444.

[17]

OpenEI. 2017. Time of Use pricing. https://openei.org/apps/USURDB/rate/view/5cbf78b25457a34e40671081#3__Energy

[18]

Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. 2017. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632 (2017).

[19]

M. Rosenstein and A. Barto. 2002. Supervised Learning Combined with an Actor-Critic Architecture TITLE2:. Technical Report. USA.

[20]

Lucas Spangher, Utkarsha Agwan, William Arnold, and Tarang Srivastava. 2021. https://github.com/utkarshapets/microgrid-RL

[21]

Lucas Spangher, Akash Gokul, Joseph Palakapilly, Utkarsha Agwan, Manan Khattar, Wann-Jiun Ma, and Costas Spanos. 2020. OfficeLearn: An OpenAI Gym Environment for Reinforcement Learning on Occupant-Level Building's Energy Demand Response. In Tackling Climate Change with Artificial Intelligence Workshop at NeurIPS, 2020.

[22]

Richard S. Sutton. 1991. Dyna, an Integrated Architecture for Learning, Planning, and Reacting. SIGART Bull. 2, 4 (July 1991), 160--163. https://doi.org/10.1145/122344.122377

Digital Library

[23]

Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.

Digital Library

[24]

José R Vázquez-Canteli and Zoltan Nagy. 2019. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied energy 235 (2019), 1072--1089.

[25]

Hao Wang and Jianwei Huang. 2016. Incentivizing energy trading for interconnected microgrids. IEEE Transactions on Smart Grid 9, 4 (2016), 2647--2657.

[26]

Yu Wang, Shiwen Mao, and R Mark Nelms. 2015. On hierarchical power scheduling for the macrogrid and cooperative microgrids. IEEE Transactions on Industrial Informatics 11, 6 (2015), 1574--1584.

Cited By

Das HLin YAgwan USpangher LDevonport AYang YDrgoňa JChong ASchiavon SSpanos C(2024)Machine Learning for Smart and Energy-Efficient BuildingsEnvironmental Data Science10.1017/eds.2023.433Online publication date: 4-Jan-2024
https://doi.org/10.1017/eds.2023.43
Jang DSpangher LSrivastava TYan LSpanos C(2023)Personalized Federated Hypernetworks for Multi-Task Reinforcement Learning in Microgrid Energy Demand ResponseProceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3600100.3623733(79-88)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3600100.3623733
Zeng YWei XYao YXu YSun HKin Victor Chan WFeng W(2023)Determining the pricing and deployment strategy for virtual power plants of peer-to-peer prosumers: A game-theoretic approachApplied Energy10.1016/j.apenergy.2023.121349345(121349)Online publication date: Sep-2023
https://doi.org/10.1016/j.apenergy.2023.121349
Show More Cited By

Index Terms

Pricing in Prosumer Aggregations using Reinforcement Learning
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
2. Hardware
  1. Power and energy
    1. Energy distribution
      1. Smart grid

Recommendations

Prosumer Pricing, Incentives and Fairness
e-Energy '19: Proceedings of the Tenth ACM International Conference on Future Energy Systems

Many current electricity rate structures for residential consumers do not provide proper incentives for consumer solar production. They also unfairly burden traditional consumers in favor of solar producing consumers ("prosumers"), by charging the bulk ...
Pricing commodities

How should a seller price her goods in a market where each buyer prefers a single good among his desired goods, and will buy the cheapest such good, as long as it is within his budget? We provide efficient algorithms that compute near-optimal prices for ...
Pricing Rule in a Clock Auction

We analyze a discrete clock auction with lowest-accepted-bid (LAB) pricing and provisional winners, as adopted by India for its 3G spectrum auction. In a perfect Bayesian equilibrium, the provisional winner shades her bid, whereas provisional losers do ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

e-Energy '21: Proceedings of the Twelfth ACM International Conference on Future Energy Systems

June 2021

528 pages

ISBN:9781450383332

DOI:10.1145/3447555

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper
Research
Refereed limited

Conference

e-Energy '21

e-Energy '21: The Twelfth ACM International Conference on Future Energy Systems

June 28 - July 2, 2021

Virtual Event, Italy

Acceptance Rates

Overall Acceptance Rate 160 of 446 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
644
Total Downloads

Downloads (Last 12 months)130
Downloads (Last 6 weeks)14

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Das HLin YAgwan USpangher LDevonport AYang YDrgoňa JChong ASchiavon SSpanos C(2024)Machine Learning for Smart and Energy-Efficient BuildingsEnvironmental Data Science10.1017/eds.2023.433Online publication date: 4-Jan-2024
https://doi.org/10.1017/eds.2023.43
Jang DSpangher LSrivastava TYan LSpanos C(2023)Personalized Federated Hypernetworks for Multi-Task Reinforcement Learning in Microgrid Energy Demand ResponseProceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation10.1145/3600100.3623733(79-88)Online publication date: 15-Nov-2023
https://dl.acm.org/doi/10.1145/3600100.3623733
Zeng YWei XYao YXu YSun HKin Victor Chan WFeng W(2023)Determining the pricing and deployment strategy for virtual power plants of peer-to-peer prosumers: A game-theoretic approachApplied Energy10.1016/j.apenergy.2023.121349345(121349)Online publication date: Sep-2023
https://doi.org/10.1016/j.apenergy.2023.121349
Tapia CUlloa DPacheco-Cunduri MHernández-Ambato JRodríguez-Flores JHerrera-Perez V(2022)Optimal Fuzzy-Based Energy Management Strategy to Maximize Self-Consumption of PV Systems in the Residential Sector in EcuadorEnergies10.3390/en1514516515:14(5165)Online publication date: 16-Jul-2022
https://doi.org/10.3390/en15145165

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten