diff --git a/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md b/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md index 6ba969f17..f666de525 100644 --- a/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md +++ b/docs/superpowers/specs/2026-04-26-distributional-rl-aggregation-design.md @@ -646,3 +646,26 @@ This contract turns the principle into an engineering invariant. New methods can **Phase 0 unit tests + Aggregation Contract: KEEP as latent infrastructure.** They were correct math validation; they neither motivated the rollout based on bad data nor depend on the val-Flat-collapse hypothesis being severe. **Plans B / C / D: PAUSED.** Files remain in `docs/superpowers/plans/` for resumption after the bug-hunt cycle on `train-f8h6q`. Resumption gate: post-fix baseline run is bug-free or all surfaced bugs are fixed. + +--- + +## Status update: ACTIVE — Plan C Phase 2 resumed 2026-04-29 + +**Triggering pathology (different from original Hold/Flat-collapse motivation):** + +`ff00af68a` (`refactor(dqn): delete vestigial RegimeConditionalDQN — MoE replaces it`) introduced a `multi_fold_convergence` regression: F0 Q-drift kill at epoch 2 (q_mean -0.077 → 0.793, ratio 10.28×), cascading into F1+F2 grad-collapse early-stop. The commit's `crates/ml/src/trainers/dqn/trainer/action.rs:107-118` change removed an `Option::None` ghost gate, activating UCB count bonuses at the GPU action selector for the first time. Verified by paired DIAG smokes (`smoke-test-qlz7t` on ff00af68a + diag fails F0; `smoke-test-wmsht` on a52d99613 + diag passes all 3 folds). + +The mechanism: UCB bonus is added to direction-branch Q-values inside `experience_action_select` before Boltzmann softmax, but the Bellman target Q computation uses raw argmax (no bonus). This selector/target asymmetry lets q_mean drift up unchecked → Q-drift kill triggers. + +**Why Thompson supersedes the bug fix:** + +UCB-as-additive-on-Q is the broken pattern. Three local fixes were considered: +- (A) Adaptive UCB coefficient via ISV decay — preserves selector/target asymmetry (just attenuates) +- (B) UCB-as-intrinsic-reward via Bellman — fixes asymmetry but adds reward-stream tuning surface +- (C) Replace eps-greedy + Boltzmann + UCB-bonus with Thompson — eliminates asymmetry by construction (Thompson samples from the learned distribution, no augment-then-argmax step) + +Option C is now Plan C Phase 2. The C51+IQN substrate already produces the distributions Thompson needs; Phase 0 tests already validate the math; eliminating the additive-on-Q exploration mechanism is a net code-surface decrease, not an addition. + +**Resumption gate satisfied:** all surfaced bugs in the original measurement-artefact hunt are fixed (commits `a86fba2b1`, `b8788511c`). The new regression introduced by `ff00af68a` is precisely the "Thompson preserves second moment, model has measurable σ asymmetry" empirical motivation referenced in §2 of the original sequencing plan — the σ asymmetry is now causing measurable training-time Q-drift, not just suboptimal selection. + +**Phase 2 execution:** plan at `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-C-phase-2.md`. Plans B / D remain on hold pending Phase 2 outcome.