spec(dqn): Plan C Phase 2 ACTIVE — UCB asymmetry regression triggers Thompson resumption

The original PAUSED state was motivated by measurement-artefact hunt
exit. The bug-hunt cycle is complete (commits a86fba2b1, b8788511c).
A new pathology has surfaced: ff00af68a's UCB count bonus activation
causes selector/target asymmetry → F0 Q-drift kill at epoch 2 →
F1+F2 cascade. Verified by paired DIAG smokes (smoke-test-qlz7t fail,
smoke-test-wmsht pass).

Thompson sampling on C51+IQN distributions eliminates the asymmetry
by construction (sample from learned distribution; no augment-then-
argmax step). Net code-surface decrease — replaces eps-greedy +
Boltzmann + UCB with one principled mechanism.

Plan C Phase 2 execution begins on branch plan-c-phase-2-thompson.
This commit is contained in:
jgrusewski
2026-04-29 16:02:53 +02:00
parent 7e2eb708a7
commit 8a535681b7

View File

@@ -646,3 +646,26 @@ This contract turns the principle into an engineering invariant. New methods can
**Phase 0 unit tests + Aggregation Contract: KEEP as latent infrastructure.** They were correct math validation; they neither motivated the rollout based on bad data nor depend on the val-Flat-collapse hypothesis being severe.
**Plans B / C / D: PAUSED.** Files remain in `docs/superpowers/plans/` for resumption after the bug-hunt cycle on `train-f8h6q`. Resumption gate: post-fix baseline run is bug-free or all surfaced bugs are fixed.
---
## Status update: ACTIVE — Plan C Phase 2 resumed 2026-04-29
**Triggering pathology (different from original Hold/Flat-collapse motivation):**
`ff00af68a` (`refactor(dqn): delete vestigial RegimeConditionalDQN — MoE replaces it`) introduced a `multi_fold_convergence` regression: F0 Q-drift kill at epoch 2 (q_mean -0.077 → 0.793, ratio 10.28×), cascading into F1+F2 grad-collapse early-stop. The commit's `crates/ml/src/trainers/dqn/trainer/action.rs:107-118` change removed an `Option::None` ghost gate, activating UCB count bonuses at the GPU action selector for the first time. Verified by paired DIAG smokes (`smoke-test-qlz7t` on ff00af68a + diag fails F0; `smoke-test-wmsht` on a52d99613 + diag passes all 3 folds).
The mechanism: UCB bonus is added to direction-branch Q-values inside `experience_action_select` before Boltzmann softmax, but the Bellman target Q computation uses raw argmax (no bonus). This selector/target asymmetry lets q_mean drift up unchecked → Q-drift kill triggers.
**Why Thompson supersedes the bug fix:**
UCB-as-additive-on-Q is the broken pattern. Three local fixes were considered:
- (A) Adaptive UCB coefficient via ISV decay — preserves selector/target asymmetry (just attenuates)
- (B) UCB-as-intrinsic-reward via Bellman — fixes asymmetry but adds reward-stream tuning surface
- (C) Replace eps-greedy + Boltzmann + UCB-bonus with Thompson — eliminates asymmetry by construction (Thompson samples from the learned distribution, no augment-then-argmax step)
Option C is now Plan C Phase 2. The C51+IQN substrate already produces the distributions Thompson needs; Phase 0 tests already validate the math; eliminating the additive-on-Q exploration mechanism is a net code-surface decrease, not an addition.
**Resumption gate satisfied:** all surfaced bugs in the original measurement-artefact hunt are fixed (commits `a86fba2b1`, `b8788511c`). The new regression introduced by `ff00af68a` is precisely the "Thompson preserves second moment, model has measurable σ asymmetry" empirical motivation referenced in §2 of the original sequencing plan — the σ asymmetry is now causing measurable training-time Q-drift, not just suboptimal selection.
**Phase 2 execution:** plan at `docs/superpowers/plans/2026-04-27-distributional-rl-thompson-plan-C-phase-2.md`. Plans B / D remain on hold pending Phase 2 outcome.