1. What is Reinforcement Learning

概述：

举个栗子：

再举一个：

2. Markov Decision Process

• Mathematical formulation of the RL problem
• Markov property: Current state completely characterises the state of the world

处理流程：

The optimal policy π*

3. Q-learning

Definitions: Value function and Q-value function：

Bellman equation：

优化策略：

Solving for the optimal policy: Q-learning

举个栗子：Playing Atari Games

Q-network Architecture

Training the Q-network: Experience Replay

Deep Q-Learning with Experience Replay

4. Policy Gradients

Intuition：

Variance reduction：

Variance reduction: Baseline

How to choose the baseline?

A better baseline: Want to push up the probability of an action from a state, if this action was better than the expected value of what we should get from that state

Actor-Critic Algorithm

5. REINFORCE 的运用

5.1 Recurrent Attention Model (RAM)

效果示意图：

5.2 AlphaGo

6. Summary

• Policy gradients: very general but suffer from high variance so requires a lot of samples.
  Challenge: sample-efficiency
• Q-learning: does not always work but when it works, usually more sample-efficient. Challenge: exploration
• Guarantees:
  Policy Gradients: Converges to a local minima of J(θ), often good enough!
  Q-learning: Zero guarantees since you are approximating Bellman equation with a complicated function approximator