Stochastic Policies in Morally Constrained (C-)SSPs
Authors: 
Charles Evans, Claire Benn, Ignacio Ojea Quintana, Pamela Robinson, Sylvie ThiébauxAuthors Info & Claims
Pages 253 - 264
Abstract
Stochastic policies often outperform deterministic ones. This is especially true for Constrained Stochastic Shortest Path (C-SSP) problems, a popular approach to planning under uncertainty with multiple objectives. Nevertheless, there are moral concerns about stochastic policies that should deter us from selecting them. In this paper, we identify some of these moral concerns and offer 'acceptability constraints' that allow only certain stochastic policies to be selected. We propose a novel C-SSP solver able to integrate our moral acceptability constraints, we evaluate its performance in a relevant test problem, and we show that our approach can successfully produce acceptable policies in morally significant domains.
Supplementary Material
This talk summarises the main findings and talking points from the paper ?Stochastic Policies in Morally Constrained (C-)SSPs.? We take a look at the Constrained Stochastic Shortest Path (C-SSP) problem framework and the potential problems that can arise when applying this framework to morally loaded planning problems. We give an overview of our approach to developing an approach to distinguishing morally acceptable policies from unacceptable policies in such problems, and demonstrate how we integrated this approach into a novel C-SSP solver algorithm. Spoken by Charles Evans on behalf of co-authors Drs. Claire Benn, Ignacio Ojea Quintana, Pamela Robinson and Sylvie Thiébaux from the Humanising Machine Intelligence project at The Australian National University.
- Download
- 136.66 MB
References
[1]
Ethan Altman. 1999. Constrained Markov Decision Processes. Chapman and Hall.
[2]
Dimitri P. Bertsekas and John N. Tsitsiklis. 1991. An Analysis of Stochastic Shortest Path Problems. Mathematics of Operations Research 16, 3 (1991), 580--595.
[3]
Vicky Charisi, Louise A. Dennis, Michael Fisher, Robert Lieck, Andreas Matthias, Marija Slavkovik, Janina Sombetzki, Alan F. T. Winfield, and Roman Yampolskiy. 2017. Towards Moral Autonomous Systems. CoRR abs/1703.04741 (2017).
[4]
Yinlam Chow and Mohammad Ghavamzadeh. 2014. Algorithms for CVaR Optimization in MDPs. In Proc. 27th Annual Conference Advances on Neural Information Processing Systems (NIPS'14). 3509--3517.
[5]
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. 2015. Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach. In Proc. 28th Annual Conference Advances on Neural Information Processing Systems (NIPS'15). 1522--1530.
[6]
Louise Dennis, Michael Fisher, Marija Slavkovik, and MattWebster. 2016. Formal verification of ethical choices in autonomous systems. Robotics and Autonomous Systems 77 (2016), 1--14.
[7]
Dmitri Dolgov and Edmund Durfee. 2005. Stationary Deterministic Policies for Constrained MDPs with Multiple Rewards, Costs, and Discount Factors. In Proc. 19th International Joint Conference on Artificial Intelligence (IJCAI'05). 1326--1331.
[8]
Eugene A. Feinberg and Adam Shwartz. 1995. Constrained Markov Decision Models with Weighted Discounted Rewards. Mathematics of Operations Research 20, 2 (1995), 302--320.
[9]
Jerzy A. Filar, Lodewijk C. M. Kallenberg, and Huey-Miin Lee. 1989. Variance-Penalized Markov Decision Processes. Mathematics of Operations Research 14, 1 (1989), 147--161.
[10]
Florian Geißer, Guillaume Povéda, Felipe Trevizan, Manon Bondouy, Florent Teichteil-Königsbuch, and Sylvie Thiébaux. 2020. Optimal and Heuristic Approaches for Constrained Flight Planning under Weather Uncertainty. In Proc. 30th International Conference on Automated Planning and Scheduling (ICAPS'20). 384--393.
[11]
Seth Lazar. 2017. Anton's Game: Deontological Decision Theory for an Iterated Decision Problem. Utilitas 29 (2017), 88--109.
[12]
Hyun-Rok Lee and Taesik Lee. 2018. Markov decision process model for patient admission decision at an emergency department under a surge demand. Flexible Services and Manufacturing Journal 30, 1 (2018), 98--122.
[13]
Hyun-Rok Lee and Taesik Lee. 2021. Multi-agent reinforcement learning algorithm to solve a partially-observable multi-agent problem in disaster response. European Journal of Operational Research 291, 1 (2021), 296--308.
[14]
Felix Lindner, Robert Mattmüller, and Bernhard Nebel. 2020. Evaluation of the moral permissibility of action plans. Artificial Intelligence 287 (2020), 103350.
[15]
Shie Mannor and John N. Tsitsiklis. 2011. Mean-Variance Optimization in Markov Decision Processes. In Proc. 28th International Conference on International Conference on Machine Learning (ICML'11). 177--184.
[16]
Samer Nashed, Justin Svegliato, and Shlomo Zilberstein. 2021. Ethically Compliant Planning within Moral Communities. In Proc. 4th AAAI/ACM Conference on AI, Ethics, and Society (AIES'21). 188--198.
[17]
Adriana Placani. 2017. When the Risk of Harm Harms. Law and Philosophy 36, 1 (2017), 77--100.
[18]
R.Tyrrell Rockafellar and Stanislav Uryasev. 2002. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance 26, 7 (2002), 1443--1471.
[19]
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. 2014. Lectures on Stochastic Programming - Modeling and Theory, Second Edition. MOS-SIAM Series on Optimization, Vol. 16. Society for Industrial and Applied Mathematics.
[20]
Justin Svegliato, Samer B. Nashed, and Shlomo Zilberstein. 2021. Ethically Compliant Sequential Decision Making. In Proc. 35th AAAI Conference on Artificial Intelligence (AAAI'21). 11657--11665.
[21]
Felipe Trevizan, Sylvie Thiébaux, and Patrik Haslum. 2017. Occupation Measure Heuristics for Probabilistic Planning. In Proc. 27th International Conference on Automated Planning and Scheduling (ICAPS'17). 306--315.
[22]
Felipe W. Trevizan, Sylvie Thiébaux, Pedro Henrique Santana, and Brian Charles Williams. 2016. Heuristic Search in Dual Space for Constrained Stochastic Shortest Path Problems. In Proc. 26th International Conference on Automated Planning and Scheduling (ICAPS'16). 326--334.
[23]
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. 2023. Reinforcement Learning in Healthcare: A Survey. Comput. Surveys 55, 1 (2023), 1--36.
Index Terms
- Stochastic Policies in Morally Constrained (C-)SSPs
Recommendations
Planning and acting in partially observable stochastic domains
In this paper, we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains. We begin by introducing the theory of Markov decision processes (mdps) and partially observable ...
Stochastic dominance in stochastic DCOPs for risk-sensitive applications
AAMAS '12: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1Distributed constraint optimization problems (DCOPs) are well-suited for modeling multi-agent coordination problems where the primary interactions are between local subsets of agents. However, one limitation of DCOPs is the assumption that the ...
Constrained Undiscounted Stochastic Dynamic Programming
In this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function. The state-space and action-sets are assumed to be finite. Constraints which are linear ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2022
939 pages
ISBN:9781450392471
DOI:10.1145/3514094
- General Chairs:
- Vincent Conitzer,
- John Tasioulas,
- Program Chairs:
- Matthias Scheutz,
- Ryan Calo,
- Martina Mara,
- Annette Zimmermann
Copyright © 2022 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 July 2022
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
AIES '22
Sponsor:
Acceptance Rates
Overall Acceptance Rate 61 of 162 submissions, 38%
Upcoming Conference
AIES '24
- Sponsor:
- sigai
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 284Total Downloads
- Downloads (Last 12 months)144
- Downloads (Last 6 weeks)16
Reflects downloads up to 02 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in