feat(dqn): Plan C A.2 — adaptive Polyak tau coupled to q-drift rate
Wires the q_drift_rate_ema kernel + ISV slot Q_DRIFT_RATE_INDEX=129 (landed infc8dbb0a8) into the production training path: - StateResetRegistry FoldReset entry isv_q_drift_rate -> 0.0 (companion to A.1's prev_epoch_q_mean reset; both ensure no cross-fold leakage of drift state). - reset_named_state dispatch arm in training_loop.rs. - Per-epoch launch_q_drift_rate_ema call after the Q-drift kill check, gated on the same `prev_epoch_q_mean.abs() > 1e-6` cold-start guard. Both q_mean_curr and q_mean_prev are in scope BEFORE the `prev_epoch_q_mean = q_mean` update, so the producer sees the correct delta. - tau_update_kernel.cu multiplies the cosine-scheduled, health-coupled tau_eff by `1 / (1 + clip(ISV[129], 0, 4))` so tau ranges [tau_base/5, tau_base] — monotone dampening under drift; healthy runs (drift ≈ 0) unaffected. - reset_for_fold comment in trainer/mod.rs notes the registry entry handles the new ISV slot. Predicted impact: Plan C fold 0 ep2 with q_mean(t)=0.82, q_mean(t-1)=-0.018, ISV[16]+ISV[21]≈0.1 -> drift_rate ≈ 8.4 -> clipped to 4 -> tau_eff = tau_base × 0.2 (5× slower target sync, dampens Q-target optimism through the inflation spike).a52d99613fold 0 with q_mean staying ~0.21 -> drift_rate ≈ 0 -> tau_eff = tau_base (unchanged). Per pearl_adaptive_moe_lambda.md — canonical "EMA-tracked diagnostic drives a controller" pattern. Per pearl_cold_path_no_exception_to_gpu_drives.md — cold-path scalar arithmetic stays on GPU. Per feedback_no_partial_refactor.md — consumer (tau_update + state_reset + producer launch) all migrate together in this commit. Layout fingerprint already shifted byfc8dbb0a8(slot + ISV_TOTAL_DIM bump); no additional fingerprint shift in this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2190,3 +2190,5 @@ Plan C Phase 2 follow-up A.2 (2026-04-29): adaptive Polyak-tau coupled to per-ep
|
||||
Predicted impact: Plan C fold 0 ep2 with `q_mean(t)=0.82, q_mean(t-1)=-0.018, ISV[16]+ISV[21]≈0.1` → drift_rate ≈ |0.84|/max(0.018, 0.1, ε) = 8.4 → clipped to 4 → tau_eff = tau_base × 0.2 (5× slower target sync, dampens Q-target optimism through inflation spike). a52d99613 fold 0 with q_mean staying ~0.21 → drift_rate ≈ 0 → tau_eff = tau_base (unchanged, healthy run unaffected).
|
||||
|
||||
Per `pearl_adaptive_moe_lambda.md` — canonical "EMA-tracked diagnostic drives a controller" pattern: kernel + ISV slot + bootstrap (0.0 cold-start) + reset (0.0 fold boundary) + observability. Per `pearl_cold_path_no_exception_to_gpu_drives.md` — even cold-path scalar arithmetic stays on GPU when its reduction inputs already live on GPU (ISV[16,21] both produced by `q_stats_kernel.cu`). Per `feedback_adaptive_not_tuned.md` — tau decay is ISV-driven, not a constant. Per `feedback_isv_for_adaptive_bounds.md` — the `4.0` upper clip and `1e-6` denom floor are structural bounds (drift saturation matching the kill-criterion's 3.0× ratio threshold; numerical-stability floor) carried in ISV slot semantics.
|
||||
|
||||
Plan C A.2 wire-up (2026-04-29, second commit): `tau_update_kernel.cu`, `state_reset_registry.rs`, `training_loop.rs::reset_named_state` + per-epoch launch site, and `trainer/mod.rs::reset_for_fold` comment all updated in lockstep with the kernel+slot landed by the previous commit. Producer launch lives between the existing Q-drift kill criterion and the `prev_epoch_q_mean = q_mean` update — both prev and curr q_means are in scope, and the `prev_epoch_q_mean.abs() > 1e-6` cold-start gate matches the kill check's gate. Tau consumer applies `1/(1+clip(ISV[129],0,4))` AFTER the existing `[tau_final, tau_base]` clamp so dampening can drive tau below tau_final on runaway drift (intent: tau_final is the design floor for healthy runs, but a runaway requires further suppression). Defensive `fminf(...,4.0f)` in the consumer guards against producer-side bugs even though the producer already clips (Invariant 1 carve-out, belt-and-braces).
|
||||
|
||||
Reference in New Issue
Block a user