总结

强化学习脉络思维导图

方法对比矩阵

Method	Model-based	Policy-based	Value-based	Actor-Critic	On-policy	Off-policy	Bootstrapping	Continuous Action	Function Approx	Stable
Policy Iteration	✅	❌	✅	❌	❌	❌	❌	❌	❌	✅
Value Iteration	✅	❌	✅	❌	❌	❌	❌	❌	❌	✅
Monte Carlo	❌	❌	✅	❌	✅	❌	❌	❌	❌	❌
TD(0)	❌	❌	✅	❌	✅	❌	✅	❌	❌	❌
SARSA	❌	❌	✅	❌	✅	❌	✅	❌	❌	❌
Q-learning	❌	❌	✅	❌	❌	✅	✅	❌	❌	❌
DQN	❌	❌	✅	❌	❌	✅	✅	❌	✅	❌
REINFORCE	❌	✅	❌	❌	✅	❌	❌	✅	✅	❌
Actor-Critic	❌	✅	✅	✅	✅	❌	✅	✅	✅	❌
A2C	❌	✅	✅	✅	✅	❌	✅	✅	✅	❌
A3C	❌	✅	✅	✅	✅	❌	✅	✅	✅	❌
PPO	❌	✅	✅	✅	✅	❌	✅	✅	✅	✅
DDPG	❌	✅	✅	✅	❌	✅	✅	✅	✅	❌
TD3	❌	✅	✅	✅	❌	✅	✅	✅	✅	❌
SAC	❌	✅	✅	✅	❌	✅	✅	✅	✅	✅

核心公式速查表

1. Dynamic Programming (Model-based)

Policy Evaluation

$$v(s) \leftarrow \sum_a \pi(a|s)\Big(r(s,a) + \gamma \sum_{s'} P(s'|s,a)\, v(s')\Big)$$

Value Iteration

$$v(s) \leftarrow \max_a \Big(r(s,a) + \gamma \sum_{s'} P(s'|s,a)\, v(s')\Big)$$

---

2. Monte Carlo

MC Policy Evaluation

$$Q(s,a) \leftarrow Q(s,a) + \alpha\big(G_t - Q(s,a)\big)$$

---

3. Temporal Difference

TD(0)

$$V(s_t) \leftarrow V(s_t) + \alpha \Big(r_t + \gamma V(s_{t+1}) - V(s_t)\Big)$$

n-step TD

$$V(s_t) \leftarrow V(s_t) + \alpha \Big(G_t^{(n)} - V(s_t)\Big)$$

---

4. Control with TD

SARSA (On-policy)

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \Big(r_t + \gamma Q(s_{t+1},a_{t+1}) - Q(s_t,a_t)\Big)$$

Q-learning (Off-policy)

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \Big(r_t + \gamma \max_a Q(s_{t+1},a) - Q(s_t,a_t)\Big)$$

Expected SARSA

$$Q \leftarrow Q + \alpha \Big(r + \gamma \mathbb{E}_{a' \sim \pi}[Q(s',a')] - Q\Big)$$

---

5. Policy Gradient

REINFORCE

$$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(a_t|s_t)\, G_t$$

---

6. Actor-Critic

Critic (TD Error)

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

Actor

$$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi(a_t|s_t)\, \delta_t$$

---

7. Advantage Actor-Critic (A2C)

Advantage

$$A(s,a) = Q(s,a) - V(s)$$

Actor Update

$$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi(a|s)\, A(s,a)$$

---

8. Deep Q-Network (DQN)

Loss

$$L(\theta) = \Big(r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta)\Big)^2$$

---

9. PPO (Clipped Objective)

Probability Ratio

$$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$$

Objective

$$L(\theta) = \mathbb{E}\Big[\min(r_t(\theta)A_t,\; \text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t)\Big]$$

---

10. SAC (Maximum Entropy RL)

Soft Q Target

$$y = r + \gamma \Big(V(s') - \alpha \log \pi(a'|s')\Big)$$

Q Loss

$$J_Q = \mathbb{E}\big[(Q(s,a) - y)^2\big]$$

Policy Loss

$$J_\pi = \mathbb{E}\big[\alpha \log \pi(a|s) - Q(s,a)\big]$$

工程技巧核心公式表

1. Target Network

Purpose

• Stabilize bootstrapping

Formula

$$y = r + \gamma \max_{a'} Q(s', a'; \theta^-)$$

Where:

• $\theta^-$ = slow-moving target parameters

---

2. Experience Replay

Purpose

• Break correlation
• Improve sample efficiency

Update (same as Q-learning)

$$Q(s,a) \leftarrow Q(s,a) + \alpha \big(y - Q(s,a)\big)$$

Where:

$$(s,a,r,s') \sim \mathcal{D}$$

---

3. Advantage Normalization

Purpose

• Reduce variance
• Stabilize updates

Formula

$$\hat{A}_t = \frac{A_t - \mu_A}{\sigma_A + \epsilon}$$

---

4. Generalized Advantage Estimation (GAE)

TD Error

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

GAE

$$A_t^{(\lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$$

---

5. Entropy Bonus

Purpose

• Encourage exploration

Objective

$$J(\theta) = J_{\text{RL}}(\theta) + \beta \mathcal{H}(\pi(\cdot|s))$$

Where:

$$\mathcal{H}(\pi) = -\sum_a \pi(a|s) \log \pi(a|s)$$

---

6. Clipped Objective (PPO)

Probability Ratio

$$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$$

Clipped Loss

$$L(\theta) = \mathbb{E}\Big[\min(r_t A_t,\; \text{clip}(r_t, 1-\epsilon, 1+\epsilon) A_t)\Big]$$

---

7. Soft Target Update

Purpose

• Prevent divergence

Formula

$$\theta^- \leftarrow \tau \theta + (1-\tau)\theta^-$$

---

8. Reward Normalization

Purpose

• Stabilize learning

Formula

$$\hat{r}_t = \frac{r_t - \mu_r}{\sigma_r + \epsilon}$$

---

9. Value Clipping (PPO)

Formula

$$V_{\text{clip}} = V_{\text{old}} + \text{clip}(V - V_{\text{old}}, -\epsilon, \epsilon)$$

---

10. Double Q-learning

Purpose

• Reduce overestimation

Formula

$$y = r + \gamma Q_{\theta^-}(s', \arg\max_a Q_\theta(s',a))$$