feat(alpha): Munchausen DQN target term kernel (Vieillard et al. 2020)
Phase E.1 Task 10. Standalone target-augmentation kernel: m = α_m · max(τ · log π(a|s), log_clip_min) V_soft(s') = max(Q_next) + τ · log Σ exp((Q_next − max) / τ) target = r + m + γ · V_soft(s') (terminal: r + m) π(a|s) ∝ exp(Q_online(s, a) / τ) — softmax policy from the online net. Munchausen bonus is implicit KL regularisation between successive policies; soft-V replaces the hard max bootstrap with a τ-weighted softmax average. Both softmaxes are computed via log-sum-exp with the max-trick. This is essential at τ ≈ 0.03 where raw exp(Q/τ) would overflow f32 for any Q-spread > 25 nats. The kernel is one-thread-per-batch-sample, no atomicAdd, no host branches. α_m, τ, log_clip_min are kernel args (not hard-coded), so a Phase E.2+ controller can ISV-drive them. Typical Vieillard values: α_m=0.9, τ=0.03, log_clip_min=-1.0. Does NOT touch any ISV slot — pure target augmentation. Cubin: target/release/build/ml-*/out/phase_e_munchausen_target.cubin (12.8 KB). Launcher integration is Task 11 (consumes target_out where the C51/MSE loss kernels currently consume `r + γ · max_a' Q_target`). Audit doc docs/isv-slots.md updated per Invariant 7.
This commit is contained in:
@@ -1595,6 +1595,12 @@ fn main() {
|
||||
// smooths into ISV[539..543) — see alpha_isv_slots.rs. Slot 547/548
|
||||
// (random baseline mean/std) are read directly from ISV by index.
|
||||
"phase_e_kill_criteria.cu",
|
||||
// Phase E.1 Task 10 (2026-05-15): Munchausen DQN target term per
|
||||
// Vieillard et al. 2020. Adds an implicit KL-regularisation bonus
|
||||
// m = α_m · max(τ · log π(a|s), log_clip_min) to the TD target,
|
||||
// plus a soft-V bootstrap V_soft(s') = τ · log Σ exp(Q/τ).
|
||||
// One thread per batch sample; log-sum-exp-stable softmax.
|
||||
"phase_e_munchausen_target.cu",
|
||||
];
|
||||
|
||||
// ALL kernels get common header (BF16 types + wrappers)
|
||||
|
||||
Reference in New Issue
Block a user