Observable subspace solution for irreducible POMDPs with infinite horizon

Cyber--attacks on critical infrastructure have two main properties: (i) they are adversarial processes, and (ii) defenders will not have access to all the information they need. We, therefore, model these problems as discrete-time infinite-horizon partially observable Markov decision processes (POMDPs) with undiscounted average payoff. Our solution creates a model of the observable subspace of the original POMDP, and then finds the control policy of the observable system. This requires neither a priori information nor belief state update. We compare our approach with the currently used value iteration approximation methods. The proposed approach reduces computational overhead, and provides better solutions under certain conditions. Furthermore, the long-run average payoff obtained by our method is predictable, which helps the controller to determine whether the algorithm performance at an acceptable level before implementation.