gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(alpha): stabilize alpha_dqn_h600_smoke — reward norm + target net + grad clip

Browse Source 
Three stabilizers applied to the H=600 DQN smoke after initial run showed
unstable training (early_mvmt=2268× at lr=1e-6, NaN at lr=1e-4):

  1. Reward normalization (--reward-scale, default 1000)
     Rewards divided by scale BEFORE the Munchausen target. TD error
     drops from ~1000 (raw reward magnitude at H=600) into O(1) target /
     gradient / weight-update scale. Action selection + rollout-R
     reporting use ORIGINAL rewards (so rvr math stays correct against
     the Task 7c baseline).

  2. Target network (--target-update-every, default 10 episodes)
     Separate w_target_dev / b_target_dev buffers. Q_next(s') forward
     uses target weights; SGD updates online only. Hard-update copies
     online → target every K episodes. Breaks the V_soft(s') chase-its-
     own-tail divergence of online-only Munchausen.

  3. Gradient clipping (--grad-clip, default 1.0)
     New `alpha_clip_inplace_kernel` in alpha_linear_q.cu (element-wise
     clamp). Applied to dW and db after grad, before SGD. Safety net.

Diagnostic fix: weight_norm was direction-insensitive — orthogonal
rotations don't change ||W||_F, so early_mvmt read ≈0 even when training.
Switched to weight_distance_from_init = ||W_now − W_init||_F +
||b_now − b_init||_F (captures rotation). q_early = q_init + distance
so kernel's |q_early − q_init| / |q_init| ratio = distance / ||W_init||_F.

With lr bumped back up to 1e-4 (default for the stabilized config),
verified at horizon=100, n_episodes=200:

  Q_SPREAD_EMA         = 23.64   (≥ 0.05)     PASS
  ACTION_ENTROPY_EMA   = 1.86    (≥ 1.0986)   PASS
  RETURN_VS_RANDOM_EMA = +0.586  (≥ 0.0)      PASS
  EARLY_Q_MOVEMENT_EMA = 0.0212  (≥ 0.01)     PASS
  Overall: PASS (H=6000 scale-up VIABLE)

early_mvmt grew monotonically (0.005 → 0.021) across the 200-episode
run — direction-sensitive diagnostic confirms genuine policy learning.

Audit doc docs/isv-slots.md updated per Invariant 7.

Next: H=600 / 1000-episode run on full data; if PASS holds, Task 13
(H=6000 scale-up) unlocks.
This commit is contained in:
jgrusewski
2026-05-15 15:57:24 +02:00
parent fa30c2dd66
commit 8958637c77
4 changed files with 194 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/isv-slots.md
View File

@@ -697,3 +697,26 @@ cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
```
This is the *runnable Task 12 deliverable*. The full H=6000 scale-up (Task 13) is the next milestone gated on a more stable run at H=600.
## Phase E.1 Task 12 stabilization (2026-05-15)
Added three stabilizers to `alpha_dqn_h600_smoke.rs` after the initial smoke showed unstable training (`early_mvmt = 2268×` at lr=1e-6, NaN at lr=1e-4):
1. **Reward normalization** (`--reward-scale`, default 1000): rewards divided by scale before being fed to the Munchausen target. Brings TD error from ~1000 (raw reward magnitude) into O(1). Action selection and rollout-R reporting use ORIGINAL rewards — only the bootstrap target uses normalized.
2. **Target network** (`--target-update-every`, default 10 episodes): separate `w_target_dev` and `b_target_dev` buffers. Q_next(s') forward uses target weights; SGD updates only the online weights. Hard-update copies online → target every K episodes. Breaks the V_soft(s') chase-its-own-tail divergence loop characteristic of online-only Munchausen.
3. **Gradient clipping** (`--grad-clip`, default 1.0): new `alpha_clip_inplace_kernel` in `alpha_linear_q.cu` (element-wise clamp). Applied to dW and db after grad kernel, before SGD step. Safety net for gradient bursts that survive the other stabilizers.
Also fixed the diagnostic: `weight_norm` was direction-insensitive (orthogonal rotations don't change `||W||_F`), so `early_mvmt` read ≈0 even when training. Switched to `weight_distance_from_init`: `||W_now − W_init||_F + ||b_now − b_init||_F`. The Rust side then sets `q_early = q_init + distance`, so the kernel's `|q_early − q_init| / |q_init|` ratio captures relative weight-space distance from init.
**Verified at horizon=100, n_episodes=200, lr=1e-4 (with all stabilizers on)**:
- Q_SPREAD_EMA = 23.64 (≥ 0.05) PASS
- ACTION_ENTROPY_EMA = 1.86 (≥ 1.0986) PASS
- RETURN_VS_RANDOM_EMA = +0.586 (≥ 0.0) PASS
- EARLY_Q_MOVEMENT_EMA = 0.0212 (≥ 0.01) PASS
- **Overall: PASS (H=6000 scale-up VIABLE)**
`early_mvmt` grew monotonically (0.005 → 0.021) — direction-sensitive diagnostic confirms genuine policy learning across episodes.
Next: tune for the real H=600 / 1000-episode run; if PASS holds, proceed to Task 13 (H=6000 scale-up).