On the Analysis of Parameter Convergence for Temporal Difference Learning of an Exemplar Balance Problem

Bipedal walking/locomotion is a challenging control problem but also an interesting problem for studying learning algorithms. In 1981, Barto and Sutton developed a RL method based on TD which used the concept of learning from failure. Moreover, over the last few years the poor/slow convergence issues has gained more attention by researchers [1]. In this paper, a closed form value function solution for an unstable plant and optimal polynomial basis for the value function are presented. The linear TD(0) algorithm is stated and it is shown that the finite horizon effect which is due to repeatedly simulating the system over a finite horizon introduces a near singularity/bias in the parameter estimation process. A method is proposed to overcome this problem. Finally, the simulation results for the exemplar problem are presented, and the parameter convergence is analyzed.

Authors and Affiliations

1. Control Systems Group, School of Electrical and Electronic Engineering, The University of Manchester, Manchester, UK

   Martin Brown & Onder Tutsoy

Editors and Affiliations

1. Department of Automatic Control and Systems Engineering, The University of Sheffield, Mappin Street, S1 3JD, Sheffield, UK

   Roderich Groß

2. Materials and Engineering Research Institute, Sheffield Hallam University, City Campus, Horward Street, S1 1WB, Sheffield, UK

   Lyuba Alboul  & Jacques Penders  & 

3. Department of Computing, Engineering and Mathematical Sciences, Bristol Robotics Laboratory (BRL), University of Bristol and the West of England, Bristol Business Park, BS16 1QD, Bistol, UK

   Chris Melhuish

4. Department of Electrical and Electronic Engineering, Imperial College London, Exhibition Road, SW7 2BT, London, UK

   Mark Witkowski

5. Department of Psychology, The University of Sheffield, Wstern Bank, S10 2TN, Sheffield, UK

   Tony J. Prescott

Reprints and permissions

© 2011 Springer-Verlag Berlin Heidelberg

Policies and ethics