article

Some Monotonicity Results for Partially Observed Markov Decision Processes

Author:

William S. LovejoyAuthors Info & Claims

Operations Research, Volume 35, Issue 5

Pages 736 - 743

Published: 01 October 1987 Publication History

Abstract

This paper provides sufficient conditions for the optimal value in a discrete-time, finite, partially observed Markov decision process to be monotone on the space of state probability vectors ordered by likelihood ratios. The paper also presents sufficient conditions for the optimal policy to be monotone in a simple machine replacement problem, and, in the general case, for the optimal policy to be bounded from below by an easily calculated monotone function.

References

[1]

AOKI, M. 1965. Optimal Control of Partially Observable Markovian Systems. J. Franklin Inst. 280: 367-386.

[2]

ALBRIGHT, S. C. 1979. Structural Results for Partially Observable Markov Decision Processes. Opns. Res. 27: 1041-1053.

Digital Library

[3]

ASTROM, K. J. 1965. Optimal Control of Markov Processes with Incomplete State Information. J. Math. Anal.Appl. 10: 174-205.

[4]

BERTSEKAS, D. 1976. Dynamic Programming and Stochastic Control. Academic Press, New York.

[5]

BLACKWELL, D. 1965. Discounted Dynamic Programming. Ann. Math. Stat. 36: 226-235.

[6]

KARLIN, S., AND Y. RINOTT. 1980. Classes of Orderings of Measures and Related Correlation Inequalities, I. Multivariate Totally Positive Distributions. J. Multivar. Anal. 10:467-498.

[7]

LOVEJOY, W. 1987. Ordered Solutions For Dynamic Programs. Math. OR 12, 269-276.

Digital Library

[8]

MONAHAN, G. 1982. A Survey of Partially Observable Markov Decision Processes. Mgmt. Sci. 28: 1-16.

Digital Library

[9]

ROSENFIELD, D. 1976. Markovian Deterioration with Uncertain Information. Opns. Res. 24: 141-155.

Digital Library

[10]

Ross, S. 1971. Quality Control Under Markovian Deterioration. Mgmt. Sci. 17: 587-596.

[11]

SMALLWOOD, R., AND E. SONDIK. 1973. The Optimal Control of Partially Observable Markov Processes over a Finite Horizon. Opns. Res. 21: 1071-1088.

Digital Library

[12]

STOYAN, D. 1983. Comparison Methods for Queues and other Stochastic Models. John Wiley & Sons, New York.

[13]

TOPKIS, D. 1978. Minimizing a Submodular Function on a Lattice. Opns. Res. 26: 305-321.

Digital Library

[14]

WHITE, C. 1979. Optimal Control-limit Strategies for a Partially Observed Replacement Problem. Int. J. Syst. Sci. 10: 321-331.

[15]

WHITE, C. 1980a. Structured Policy Results for a Single Stage Decisionmaking Under Uncertainty. IEEE Trans. Syst. Man Cybernet. 10: 891-894.

[16]

WHITE, C. 1980ft. Monotone Control Laws for Noisy, Countable-state Markov Chains. Eur. J. Opns. Res. 5: 124-132.

[17]

WHITT, W. 1979. A Note on the Influence of the Sample on the Posterior Distribution. J. Am. Stat. Assoc. 74: 424-426.

[18]

WHITT, W. 1982. Multivariate Monotone Likelihood Ratio and Uniform Conditional Stochastic Order. J. Appl. Prob. 19: 695-701.

Cited By

Larrañaga MAssaad MDestounis APaschos G(2021)Dynamic pilot allocation over Markovian fading channels: A restless bandit approach2016 IEEE Information Theory Workshop (ITW)10.1109/ITW.2016.7606842(290-294)Online publication date: 11-Mar-2021
https://dl.acm.org/doi/10.1109/ITW.2016.7606842
Yao GBedewy AShroff N(2021)Age-Optimal Low-Power Status Update over Time-Correlated Fading Channel2021 IEEE International Symposium on Information Theory (ISIT)10.1109/ISIT45174.2021.9517880(2972-2977)Online publication date: 12-Jul-2021
https://dl.acm.org/doi/10.1109/ISIT45174.2021.9517880
Meshram RKaza K(2021)Indexability and Rollout Policy for Multi-State Partially Observable Restless Bandits2021 60th IEEE Conference on Decision and Control (CDC)10.1109/CDC45484.2021.9683132(2342-2347)Online publication date: 14-Dec-2021
https://dl.acm.org/doi/10.1109/CDC45484.2021.9683132
Show More Cited By

Index Terms

Some Monotonicity Results for Partially Observed Markov Decision Processes

Recommendations

Solution Procedures for Partially Observed Markov Decision Processes

We present three algorithms to solve the infinite horizon, expected discounted total reward partially observed Markov decision process POMDP. Each algorithm integrates a successive approximations algorithm for the POMDP due to A. Smallwood and E. Sondik ...
Computationally Feasible Bounds for Partially Observed Markov Decision Processes

<P>A partially observed Markov decision process POMDP is a sequential decision problem where information concerning parameters of interest is incomplete, and possible actions include sampling, surveying, or otherwise collecting additional information. ...
Partially Observable Risk-Sensitive Markov Decision Processes

We consider the problem of minimizing a certainty equivalent of the total or discounted cost over a finite and an infinite time horizon that is generated by a partially observable Markov decision process POMDP. In contrast to a risk-neutral decision ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Operations Research

Operations Research Volume 35, Issue 5

October 1987

157 pages

ISSN:0030-364X

Issue’s Table of Contents

Publisher

INFORMS

Linthicum, MD, United States

Publication History

Published: 01 October 1987

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Larrañaga MAssaad MDestounis APaschos G(2021)Dynamic pilot allocation over Markovian fading channels: A restless bandit approach2016 IEEE Information Theory Workshop (ITW)10.1109/ITW.2016.7606842(290-294)Online publication date: 11-Mar-2021
https://dl.acm.org/doi/10.1109/ITW.2016.7606842
Yao GBedewy AShroff N(2021)Age-Optimal Low-Power Status Update over Time-Correlated Fading Channel2021 IEEE International Symposium on Information Theory (ISIT)10.1109/ISIT45174.2021.9517880(2972-2977)Online publication date: 12-Jul-2021
https://dl.acm.org/doi/10.1109/ISIT45174.2021.9517880
Meshram RKaza K(2021)Indexability and Rollout Policy for Multi-State Partially Observable Restless Bandits2021 60th IEEE Conference on Decision and Control (CDC)10.1109/CDC45484.2021.9683132(2342-2347)Online publication date: 14-Dec-2021
https://dl.acm.org/doi/10.1109/CDC45484.2021.9683132
Kaza KMehta VMeshram RMerchant S(2018)Restless bandits with cumulative feedback: Applications in wireless networks2018 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC.2018.8377345(1-6)Online publication date: 15-Apr-2018
https://dl.acm.org/doi/10.1109/WCNC.2018.8377345
Liu JWilliams R(2018)Optimal Intermittent Deployment and Sensor Selection for Environmental Sensing with Multi-Robot Teams2018 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA.2018.8460215(1078-1083)Online publication date: 21-May-2018
https://dl.acm.org/doi/10.1109/ICRA.2018.8460215
Habachi OHayel YEl-Azouzi R(2018)Optimal energy-delay tradeoff for opportunistic spectrum access in cognitive radio networksTelecommunications Systems10.1007/s11235-017-0370-867:4(763-780)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1007/s11235-017-0370-8
Smith JUlu C(2017)Risk Aversion, Information Acquisition, and Technology AdoptionOperations Research10.5555/3216622.321663365:4(1011-1028)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.5555/3216622.3216633
Krishnamurthy VPareek U(2015)Myopic Bounds for Optimal Policy of POMDPsOperations Research10.5555/3215716.321573163:2(428-434)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.5555/3215716.3215731
Krishnamurthy VPareek U(2015)Myopic Bounds for Optimal Policy of POMDPsOperations Research10.5555/3215696.321571163:2(428-434)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.5555/3215696.3215711
Krishnamurthy VPareek U(2015)Myopic Bounds for Optimal Policy of POMDPsOperations Research10.5555/3215661.321567462:2(428-434)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.5555/3215661.3215674
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents