Abstract
In the last decades, control problems with infinite horizons and discount factors have become increasingly central not only for economics but also for applications in artificial intelligence and machine learning. The strong links between reinforcement learning and control theory have led to major efforts toward the development of algorithms to learn how to solve constrained control problems. In particular, discount plays a role in addressing the challenges that come with models that have unbounded disturbances. Although algorithms have been extensively explored, few results take into account time-dependent state constraints, which are imposed in most real-world control applications. For this purpose, here we investigate feasibility and sufficient conditions for Lipschitz regularity of the value function for a class of discounted infinite horizon optimal control problems subject to time-dependent constraints. We focus on problems with data that allow nonautonomous dynamics, and Lagrangian and state constraints that can be unbounded with possibly nonsmooth boundaries.
Similar content being viewed by others
Notes
In the sense of set-valued maps, see e.g. [1]-Section 1.4.
Here \(W^{1,1}(a,b)\) stands for the space of all absolutely continuous functions on [a, b] endowed with the norm \(\left\| g \right\| =g(a)+\int _a^b g'(s)ds\).
We recall that for a function \(q\in L^1_{\textrm{loc}}([t_0,+\infty [;{\mathbb {R}})\) the integral
$$\begin{aligned} \int _{t_0}^\infty q (t) \,dt:=\lim _{T \rightarrow \infty }\int _{t_0}^T q (t) \,dt, \end{aligned}$$provided this limit exists.
References
Aubin J-P, Frankowska H (2009) Set-valued analysis. Modern Birkhäuser Classics. Birkhäuser Boston Inc, Boston, MA
Baird L (1995) Residual algorithms: reinforcement learning with function approximation. In: Machine learning proceedings. Elsevier, pp 30–37
Basco V (2022) Weak epigraphical solutions to Hamilton–Jacobi–Bellman equations on infinite horizon. J Math Anal Appl 515(2):126452
Basco V, Cannarsa P, Frankowska H (2018) Necessary conditions for infinite horizon optimal control problems with state constraints. Math Control Relat Fields 8(3–4):535–555
Basco V, Frankowska H (2019) Hamilton–Jacobi–Bellman equations with time-measurable data and infinite horizon. Nonlinear Differ Equ Appl 26(1):7
Basco V, Frankowska H (2019) Lipschitz continuity of the value function for the infinite horizon optimal control problem under state constraints. In: Alabau-Boussouira F, et al (eds) Trends in control theory and partial differential equations, vol 32 of Springer INdAM Series. Springer International Publishing, pp 15 – 52
Bertsekas D (2022) Dynamic programming and optimal control, volume 1. Athena scientific
Bertsekas D (2019) Reinforcement learning and optimal control. Athena Scientific
Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36(1):226–235
Boyan J, Moore A (1994) Generalization in reinforcement learning: safely approximating the value function. Adv Neural Inf Process Syst 7
Calafiore GC, Fagiano L (2012) Robust model predictive control via scenario optimization. IEEE Trans Autom Control 58(1):219–224
Cannon M, Kouvaritakis B, Raković SV, Cheng Q (2010) Stochastic tubes in model predictive control with probabilistic constraints. IEEE Trans Autom Control 56(1):194–200
Crandall MG, Evans LC, Lions P-L (1984) Some properties of viscosity solutions of Hamilton–Jacobi equations. Trans Amer Math Soc 282(2):487–502
Crandall MG, Lions P-L (1983) Viscosity solutions of Hamilton–Jacobi equations. Trans Amer Math Soc 277(1):1–42
De Jager B, Van Keulen T (2013) Optimal control of hybrid vehicles. Springer, Kessels
De Pinho MR, Foroozandeh Z, Matos A (2016) Optimal control problems for path planing of AUV using simplified models. In: 2016 IEEE 55th conference on decision and control (CDC), pp 210–215. IEEE
Farina M, Giulioni L, Magni L, Scattolini R (2013) A probabilistic approach to model predictive control. In: 52nd IEEE conference on decision and control. IEEE, pp 7734–7739
Feichtinger G, Kovacevic RM, Tragler G (2018) Control systems and mathematical methods in economics, volume 687. Lecture Notes in Economics and Mathematical Systems. Springer
Frankel A (2016) Discounted quotas. J Econ Theory 166:396–444
Gordon GJ (1995) Stable function approximation in dynamic programming. In: Machine learning proceedings. Elsevier, pp 261–268
Hashimoto T (2013) Probabilistic constrained model predictive control for linear discrete-time systems with additive stochastic disturbances. In: 52nd IEEE conference on decision and control. IEEE, pp 6434–6439
Hewing L, Zeilinger MN (2018) Stochastic model predictive control for linear systems using probabilistic reachable sets. In: 2018 IEEE conference on decision and control (CDC), pp 5182–5188. IEEE
Kamgarpour M, Summers T (2017) On infinite dimensional linear programming approach to stochastic control. IFAC-PapersOnLine 50(1):6148–6153
Kouvaritakis B, Cannon M, Couchman P (2006) Mpc as a tool for sustainable development integrated policy assessment. IEEE Trans Autom Control 51(1):145–149
Kouvaritakis B, Cannon M, Raković SV, Cheng Q (2010) Explicit use of probabilistic distributions in linear predictive control. Automatica 46(10):1719–1724
Margellos K, Goulart P, Lygeros J (2014) On the road between robust optimization and the scenario approach for chance constrained optimization problems. IEEE Trans Autom Control 59(8):2258–2263
Menon PKA, Briggs MM (1990) Near-optimal midcourse guidance for air-to-air missiles. J Guid Control Dyn 13(4):596–602
Munos R (1998) A general convergence method for reinforcement learning in the continuous case. In: European conference on machine learning. Springer, pp 394–405
Munos R (2000) A study of reinforcement learning in the continuous case by the means of viscosity solutions. Mach Learn 40:265–299
Nystrup P, Boyd S, Lindström E, Madsen H (2019) Multi-period portfolio selection with drawdown control. Ann Oper Res 282(1):245–271
Postoyan R, Buşoniu L, Nešić D, Daafouz J (2016) Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. IEEE Trans Autom Control 62(6):2736–2749
Schildbach G, Goulart P, Morari M (2015) Linear controller design for chance constrained systems. Automatica 51:278–284
Van Parys BPG, Goulart PJ, Morari M (2013) Infinite horizon performance bounds for uncertain constrained systems. IEEE Trans Autom Control 58(11):2803–2817
Vinter RB (2000) Optimal Control. Birkhäuser, Boston, MA
Weston J, Tolić D, Palunko I (2022) Mixed use of Pontryagin’s principle and the Hamilton–Jacobi–Bellman equation in infinite-and finite-horizon constrained optimal control. In: International conference on intelligent autonomous systems. Springer, pp 167–185
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares no conflicts of interest in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Basco, V. Control problems on infinite horizon subject to time-dependent pure state constraints. Math. Control Signals Syst. 36, 423–450 (2024). https://doi.org/10.1007/s00498-023-00372-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00498-023-00372-3