feat(alpha): Munchausen DQN target term kernel (Vieillard et al. 2020)

Phase E.1 Task 10. Standalone target-augmentation kernel:

  m            = α_m · max(τ · log π(a|s), log_clip_min)
  V_soft(s')   = max(Q_next) + τ · log Σ exp((Q_next − max) / τ)
  target       = r + m + γ · V_soft(s')   (terminal: r + m)

π(a|s) ∝ exp(Q_online(s, a) / τ) — softmax policy from the online net.
Munchausen bonus is implicit KL regularisation between successive policies;
soft-V replaces the hard max bootstrap with a τ-weighted softmax average.

Both softmaxes are computed via log-sum-exp with the max-trick. This is
essential at τ ≈ 0.03 where raw exp(Q/τ) would overflow f32 for any
Q-spread > 25 nats. The kernel is one-thread-per-batch-sample, no
atomicAdd, no host branches.

α_m, τ, log_clip_min are kernel args (not hard-coded), so a Phase E.2+
controller can ISV-drive them. Typical Vieillard values: α_m=0.9, τ=0.03,
log_clip_min=-1.0.

Does NOT touch any ISV slot — pure target augmentation.

Cubin: target/release/build/ml-*/out/phase_e_munchausen_target.cubin (12.8 KB).

Launcher integration is Task 11 (consumes target_out where the C51/MSE
loss kernels currently consume `r + γ · max_a' Q_target`). Audit doc
docs/isv-slots.md updated per Invariant 7.
This commit is contained in:
jgrusewski
2026-05-15 14:11:00 +02:00
parent d493b729bf
commit b1ba41d403
3 changed files with 144 additions and 0 deletions

View File

@@ -1595,6 +1595,12 @@ fn main() {
// smooths into ISV[539..543) — see alpha_isv_slots.rs. Slot 547/548
// (random baseline mean/std) are read directly from ISV by index.
"phase_e_kill_criteria.cu",
// Phase E.1 Task 10 (2026-05-15): Munchausen DQN target term per
// Vieillard et al. 2020. Adds an implicit KL-regularisation bonus
// m = α_m · max(τ · log π(a|s), log_clip_min) to the TD target,
// plus a soft-V bootstrap V_soft(s') = τ · log Σ exp(Q/τ).
// One thread per batch sample; log-sum-exp-stable softmax.
"phase_e_munchausen_target.cu",
];
// ALL kernels get common header (BF16 types + wrappers)