Course Title and Code

Reinforcement Learning Based Control (EE5531)

Course Credit

3-0-0-3 (Lecture-Tutorial-Practical-Credit)

Course Category

Elective

Target Programme

UG and PG

Target Discipline

EE primarily

Prerequisite

Knowledge of state space representation and probability

Learning Outcomes

At the end of the course, students should be able to understand the learning-based control methods and should be able to formulate an on-policy or off-policy RL algorithm suitably to find the control even without complete knowledge of the system dynamics.

Course Content

S. No. Topic Lecture Hours
1 Introduction: Overview of applications of Reinforcement Learning (RL) in control: aerobatic helicopter flight, quadrotor control, pure pursuit control. 3
2 Introduction to state space representation, linear quadratic regulation, linear quadratic tracking problem, general nonlinear optimal control problem formulation, Hamilton Jacobi Bellman (HJB) equation, dynamic programming. 6
3 Finite Markov Decision Processes: The agent-environment interface, policies and value functions, optimal policies and optimal value functions, Bellman optimality equations, value iteration, policy iterations. 9
4 Temporal difference methods: SARSA, Q learning, n step boot strapping, actor-critic methods, Monte Carlo methods. 6
5 Solution of Optimal regulation and tracking: Offline Policy iteration, Online actor-critic approximators, Q-learning algorithm for algebraic Ricatti Equations, On-policy and off-policy RL for H infinity control. 9
6 Case studies of RL-based control: Control of autonomous guided vehicle with optimal speed, Cart-pole balancing, Control of a quadrotor. 9
Total: 42 hours

Textbooks

  • Richard S. Sutton and Andrew G. Barto, “Introduction to Reinforcement Learning,” 2nd Edition, MIT Press, 2017. ISBN-13 978-0262039246.
  • Dimitri Bertsekas, “Reinforcement Learning and Optimal Control,” Athena Scientific; 1st edition, 2019. ISBN-13 978-1886529397.

References

  • Hwangbo, Jemin, et al. "Control of a quadrotor with reinforcement learning." IEEE Robotics and Automation Letters 2.4 (2017): 2096-2103.
  • Kiumarsi, Bahare, et al. "Optimal and autonomous control using reinforcement learning: A survey." IEEE Transactions on Neural Networks and Learning Systems 29.6 (2017): 2042-2062.