diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md b/docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md new file mode 100644 index 000000000..e08846fdc --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md @@ -0,0 +1,227 @@ +# Track 3 — Controllers V7 Audit Triage + +**Phase:** Phase 1 — Track 3 (Adaptive controllers audit) +**Captured against commit:** `5c70c68a1` +**Hardware:** RTX 3050 Ti Laptop GPU (4 GB) — **preliminary smoke-scale verdicts**; L40S validation pending +**Data source:** `controller_activity.rs` smoke run (3 folds × 20 epochs = 60 HEALTH_DIAG boundaries, `dqn-smoketest` profile, 176k-bar fxcache) from the Phase 0 baseline capture. +**Fire-detection semantic:** intervention-based, per commit `ed4b30b49`. A controller "fires" only if it made an *adaptive intervention* that epoch — not merely because its observable output ticked. See `training_loop.rs:2431–2491` for the per-controller rules. +**Log ref:** `/tmp/foxhunt_smoke/controller_activity.log` (local; not committed). + +--- + +## Executive summary + +V7 triage across the 7 controllers named in spec §5.3. Cumulative fire counts across the 60 epochs of the smoke run (final `[CTRL_FIRE]` summary line, `controller_activity.log:2605`): + +``` +[CTRL_FIRE] anti_lr=0.000 tau=0.033 gamma=0.017 clip=0.200 cql=0.033 cost=0.000 +``` + +Per fire count verified by re-tokenising the log: `anti_lr=0/60, tau=2/60, gamma=1/60, clip=12/60, cql=2/60, cost=0/60`. + +- **0 LOAD-BEARING** — no controller fires in > 50 % of epochs. The policy is not *currently* held on the rails by adaptive intervention. +- **6 DIAGNOSTIC** — C1 (anti_lr), C2 (tau), C3 (gamma), C5 (cql_alpha), C6 (cost_anneal), and C7 (base LR scheduler) all fall below the 10 % diagnostic threshold at smoke scale. +- **1 CANDIDATE FOR DELETE (pending ablation)** — C4 (grad_clip) fires in 20 % of epochs (12/60). Above diagnostic threshold, below load-bearing. Ablation needed to settle whether removing it loses real work or is within noise. +- **0 ROOT-CAUSE FIXES** — no controller is presently firing as a *symptom of an upstream bug* (smoke-scale; this can change at L40S once the magnitude branch actually learns and health fluctuates). + +**Production-readiness read**: per the V7 framing, the absence of load-bearing controllers is the desired outcome. The concerning signal is not that controllers fire too often, but that **several of them cannot fire at smoke scale at all by construction** — the smoke profile uses `LRDecayType::Constant` (so the scheduler's `get_lr()` is invariant across epochs), and anti-LR's 5-epoch warmup plus sign-thresholded gates are unlikely to ever trigger on degenerate smoke Sharpe. Their 0 % fire rate is *uninformative* at this scale rather than positive evidence of diagnostic-only behaviour. Promote each preliminary verdict to final only after L40S validation. + +--- + +## Per-controller verdicts + +### C1 — Anti-intuitive LR (`anti_lr`) + +**Current role:** epoch-level asymmetric multiplier on the scheduled LR. Short window `max(recent_sharpe)` above threshold → boost LR (escape plateau); long-window `mean(recent_sharpe)` below `−threshold` → damp LR (stabilise). Activates only after `epoch >= anti_lr_warmup = 5`. Implemented in `training_loop.rs:2911–2954`. + +**Measured firing rate:** 0 / 60 (0.000) cumulative. + +**Wiring check:** + +1. Fire detection at `training_loop.rs:2468`: `fire_lr = prev.lr.is_finite() && (cur_lr − prev.lr).abs() > 1e-10`. +2. `cur_lr = self.lr_scheduler.get_lr()` (line 2460) — the *scheduler* LR, captured **before** the anti_lr multiplier is applied (the anti_lr adjustment lives at `training_loop.rs:2912–2943`, i.e. later in the epoch than the fire-detection block at `2431–2491`). +3. The comment at 2457–2459 claims this is intentional ("the scheduler value is the one that was actually in effect"). The semantic consequence: `fire_lr` measures *scheduler stepping*, not *anti-LR intervention*. Under smoke profile `LRDecayType::Constant` (`config.rs:1456`, `dqn-smoketest.toml` specifies no override), `get_lr()` is invariant across epochs, so `fire_lr` is necessarily 0. + +**Verdict:** DIAGNOSTIC (preliminary, smoke-scale inert). + +**Wiring surprise:** the fire-detection semantic does **not** observe anti_lr's actual intervention. It observes the base scheduler. Under a constant base LR the two are indistinguishable, but under a decaying scheduler (L40S `Cosine` or `Linear`) `fire_lr` will tick *every epoch* from pure decay drift, which is exactly the false-positive class the V7 semantic was rewritten to eliminate in commit `ed4b30b49`. This is a WIRE-PROPERLY follow-up: + +- Option A: change `cur_lr` to snapshot the *post*-anti-LR value (move the capture below `training_loop.rs:2943`). Measures intervention. +- Option B: detect anti-LR by the multiplier itself (`fire_lr = anti_mult != 1.0`) — zero-cost, unambiguous, and already computed in `training_loop.rs:2936–2942`. + +Recommend Option B for Track 3 Step 2; see §Proposed Phase 2 changes below. + +**Ablation to confirm (if needed):** set `anti_lr_good_mult = 1.0` and `anti_lr_bad_mult = 1.0` in `dqn-smoketest.toml`; re-run. Expected: no detectable difference at smoke scale (anti_lr gate requires Sharpe outside ±threshold, which smoke-Sharpe rarely reaches). Cheap to verify once wiring is fixed. + +--- + +### C2 — Adaptive tau (target-net EMA rate) + +**Current role:** cosine-annealed base tau with health-coupled floor. The scheduled tau comes from `compute_cosine_annealed_tau(training_step, tau, tau_final, tau_anneal_steps)` (`fused_training.rs:1193`). It is then passed through `apply_health_coupled_tau_floor` (`gpu_dqn_trainer.rs:8205`) which raises the floor to `0.01 × (1 − health)` — i.e. during a collapse (health ≈ 0), tau is forced up to 0.01 to accelerate target propagation. The *effective* tau is cached in `last_tau_eff` and that is what fire detection reads. + +**Measured firing rate:** 2 / 60 (0.033). + +**Wiring check:** + +- Fire: `fire_tau = prev.tau.is_finite() && (cur_tau − prev.tau).abs() > 1e-6` (`training_loop.rs:2469`). +- `cur_tau = self.last_tau_eff` (`training_loop.rs:2461`), populated from `fused_ctx.last_tau_eff()` (`training_loop.rs:1954`). This is the post-floor value, so delta detection captures both cosine-decay ticks *and* health-floor interventions. +- Both of the 2 fires occur at fold-boundary epochs (`HEALTH_DIAG[0]` of folds 2 and 3), suggesting re-seeding of the training step counter flips the cosine-annealed value while the prior fold's `last_tau_eff` persists — the fold boundary is what crosses the 1e-6 delta threshold, not an intra-fold health event. + +**Verdict:** DIAGNOSTIC (3.3 % ≪ 10 %). + +**Ablation to confirm (if needed):** replace `apply_health_coupled_tau_floor` with a pass-through that returns `tau_scheduled` unchanged. Expected: identical trajectory at smoke scale (health stays near 0.5, floor of `0.01 × 0.5 = 0.005` never exceeds scheduled tau of ~0.01). If training diverges, the floor is load-bearing at this scale (would flip to LOAD-BEARING). At production scale where health swings harder, re-measure. + +--- + +### C3 — Adaptive gamma (discount) + +**Current role:** two-level controller. Host side (`training_loop.rs:647–665`): ticks `self.adaptive_gamma` by ±0.005 / ±0.01 per epoch based on atom utilisation (`util_ema`), clamped to `[0.90, 0.95]`. Device side (`gpu_dqn_trainer.rs:8221–8230`): `gamma_eff = gamma_base + 0.005·(regime_stability − 0.5) − 0.05·(1 − health)`, clamped to `[0.90, 0.995]`. `last_gamma_eff` is the post-clamp, post-correction value — what the fire detector reads. + +**Measured firing rate:** 1 / 60 (0.017). + +**Wiring check:** + +- Fire: `fire_gamma = prev.gamma.is_finite() && (cur_gamma − prev.gamma).abs() > 1e-4` (`training_loop.rs:2470`). +- `cur_gamma = self.last_gamma_eff` — the post-clamp effective value. +- Trajectory in the log: gamma transitions 0.990 → 0.900 at fold 1 epoch 1, then pins at 0.900 for the remaining 59 epochs. The host-side scheduled `adaptive_gamma` *does* walk (G4 logs show 0.905, 0.910, 0.915, 0.920, 0.925, 0.930 mid-fold) but the device-side correction `−0.05·(1 − health)` with health ≈ 0.5 subtracts ~0.025 and the clamp floor at 0.9 absorbs the difference. So `last_gamma_eff` is pinned to the floor. + +**Wiring surprise:** the effective gamma is saturated at the floor of `[0.9, 0.995]` for 59/60 epochs. That is effectively *a constant* at smoke scale, not an adaptive controller. The host-side ticks are invisible to downstream training. + +**Verdict:** DIAGNOSTIC (1.7 % ≪ 10 %). At production scale with a warmer `health` signal the floor may release and the scheduler's host-side ticks will become visible — re-measure on L40S before finalising. + +**Ablation to confirm (if needed):** bypass `apply_adaptive_gamma` entirely and pass `adaptive_gamma` straight through. Expected: identical trajectory at smoke scale (the floor dominates anyway). At L40S, the effective gamma will differ — that's the run where the controller's real contribution shows. + +--- + +### C4 — Adaptive grad-clip + +**Current role:** EMA-tracked clip threshold re-computed every training step. The *fire* semantic is **not** delta on the threshold — it is the *intervention* latch `grad_clip_kicked_this_epoch`, set in `run_training_steps_slices` iff `raw_grad_norm > adaptive_clip_value` at any step (`training_loop.rs:1677–1685`). Reset per epoch in `reset_epoch_state` (`training_loop.rs:928`). This is the commit-ed4b30b49 semantic — "load-bearing only if it actually clamped". + +**Measured firing rate:** 12 / 60 (0.200). + +**Wiring check:** + +- Set at `training_loop.rs:1683` — `gr.raw_grad_norm.is_finite() && active_clip.is_finite() && gr.raw_grad_norm > active_clip`. +- Reset at `training_loop.rs:928` in `reset_epoch_state`. +- Read once per epoch at `training_loop.rs:2472`: `fire_clip = self.grad_clip_kicked_this_epoch`. +- The 12 fires are distributed across all three folds (2 in fold 1, 4 in fold 2, 6 in fold 3), trending up toward fold 3 — consistent with the clip threshold's EMA settling tighter while per-step grad norms occasionally spike. + +**Verdict:** CANDIDATE FOR DELETE (pending ablation). + +20 % is above the DIAGNOSTIC ceiling (10 %) and below the LOAD-BEARING floor (50 %). Per §5.3 decision matrix: measure removal impact and delete if within noise. + +**Ablation to confirm:** disable adaptive clip and run with the hard fallback clip. Two variants, cheapest first: + +- **Variant 1** (~25 min): set the adaptive clip to a fixed large value (e.g. `1e6`) so it never triggers. Re-run controller_activity + multi_fold. Check: does `raw_grad_norm` ever actually exceed the hard clip at the trainer's outer-ring limit? If yes and training diverges, flip to LOAD-BEARING. If no divergence, DELETE. +- **Variant 2**: replace adaptive EMA with a fixed clip at the steady-state value the EMA converges to on a clean run (empirically ~10 at smoke scale after the `gradient_clip_norm=1` default change). Tests whether it's the *adaptiveness* that matters or the clip itself. + +Remember: even if C4 is deleted, the per-component gradient budgets (IQN=60 %, CQL=25 %, C51=10 %, Ens=5 %) remain — those are not adaptive controllers, they are a fixed allocation. + +--- + +### C5 — CQL alpha schedule + +**Current role:** CQL pessimism weight computed device-side (`gpu_dqn_trainer.rs:4905–4921`) as `cql_alpha = config.cql_alpha × (1 − regime_stability) × health`. Collapse → 0 (CQL off), volatile + healthy → full, stable + healthy → 0. `last_cql_alpha_eff` is the post-formula value. + +**Measured firing rate:** 2 / 60 (0.033). + +**Wiring check:** + +- Fire: `fire_cql = prev.cql_alpha.is_finite() && (cur_cql − prev.cql_alpha).abs() > 1e-5` (`training_loop.rs:2473`). +- `cur_cql = self.last_cql_alpha_eff` populated from `fused_ctx.last_cql_alpha_eff()` (`training_loop.rs:1949`). +- Trajectory: `cql_alpha_eff` observed values are `0.0000`, `0.0297`, `0.0348`. The HEALTH_DIAG `effective [cql_alpha=0.0000 ...]` is read on `5c70c68a1` — CQL alpha stays at 0.0000 for most epochs and only takes its non-zero values at the same fold-boundary positions as tau fires. Like C2, the 2 fires look like fold-boundary recomputation, not intra-fold adaptation. +- The formula multiplies by `(1 − regime_stability)`. At smoke scale `regime_stability` comes from ISV[11] which is largely unfilled; from `read_isv_regime` (`gpu_dqn_trainer.rs:8180`) it clamps to `[0, 1]`, most commonly 0.5 fallback. Net effect: CQL alpha is near-zero by construction because the formula gates it on a signal that isn't mature at smoke scale. + +**Verdict:** DIAGNOSTIC (3.3 % ≪ 10 %). Same caveat as C3 — at L40S, `regime_stability` will carry real information and the fire rate may rise. + +**Ablation to confirm (if needed):** replace the multiplier with a constant `cql_alpha_eff = config.cql_alpha`. Expected: no effect on training at smoke scale because the kernel sees near-zero alpha either way (smoke `config.cql_alpha` is small). At L40S this tests whether the pessimism modulation is contributing real regularisation. + +--- + +### C6 — Cost-anneal schedule + +**Current role:** deterministic sigmoid curriculum `cost_anneal_factor = 1 / (1 + exp(−(epoch − 10) / 3))` (`training_loop.rs:449`). Epoch 0 ≈ 3 %, 10 = 50 %, 20 ≈ 97 %. Multiplies transaction costs at the reward level. + +**Measured firing rate:** 0 / 60 (0.000) — **by design**. + +**Wiring check:** + +- Fire: `fire_cost = false` hard-coded at `training_loop.rs:2475`. Per the inline comment (`2451–2455`): "pure deterministic sigmoid of current_epoch. No adaptive/reactive component, so it CANNOT be load-bearing in the V7 sense. Always reported as 0 fires. Still logged so the [CTRL_FIRE] line shows the full 6-tuple." +- `cur_cost = self.cost_anneal_factor` is captured (line 2465) for the `prev_controller_values` snapshot, but `fire_cost` is never evaluated against it. + +**Verdict:** DIAGNOSTIC (excluded from load-bearing test by design, per spec §5.3). + +**Wiring note, not a finding:** the current test gates C6 at 0 by design, so 0 fires is *uninformative*. The spec's framing is correct — this is a curriculum schedule, not a closed-loop controller, and it cannot be "load-bearing in the V7 sense" because there's nothing reacting to training state. The only failure mode is "curriculum wrong" (transaction-cost ramp too fast / too slow), which is a hyperparameter question, not a controller question. No ablation required at Track 3 level; the reward-contribution audit (Track 2) is the relevant forum for questioning whether the cost-anneal curriculum is sensibly shaped. + +--- + +### C7 — Base LR scheduler + +**Current role:** epoch-level scheduled LR (called `lr_scheduler.step()` once per epoch at `training_loop.rs:2883`). Four decay strategies — `Constant`, `Linear`, `Cosine`, `Exponential` — plus optional linear warmup, defined in `crates/ml/src/trainers/dqn/lr_scheduler.rs`. Smoke profile uses `Constant` (via `DQNHyperparameters::conservative` default, `config.rs:1456`); the TOML does not override. + +**Measured firing rate:** **not measured** (spec §5.3 audit-question framing says C7 is "always active — its purpose isn't to fire"). The Phase 0 diagnostic does not emit a `controller_fire_scheduler` boolean, and `ControllerFireCounts` has no field for it (`mod.rs:193–200`). + +**Wiring check:** + +- The scheduler steps unconditionally per epoch (`training_loop.rs:2883`), regardless of training signal. Under `Constant`, `get_lr()` is invariant — zero intervention. Under `Cosine`/`Linear`/`Exponential`, `get_lr()` changes every epoch by construction. +- As noted under C1, the anti_lr fire detector reads `lr_scheduler.get_lr()` *before* the anti-LR multiplier applies, so C7's behaviour is what `fire_lr` ends up measuring whenever decay is non-constant. Under the smoke `Constant` setting these are equivalent (both 0 fires); under production settings they diverge. + +**Verdict:** DIAGNOSTIC by construction (pure open-loop curriculum, analogous to C6). Not load-bearing in the V7 sense — there is nothing to intervene. + +**Production-readiness note:** while C7 cannot be load-bearing, it does shape training. The meaningful audit question for C7 is "is the chosen decay type correct?" — a hyperparameter choice, not a controller-firing question. Out of scope for Track 3. + +**No ablation proposed.** The test for C7 is "does a constant LR converge as well as a decayed LR?" — that belongs in hyperparameter tuning, not the controller-load-bearing audit. + +--- + +## Cross-controller synthesis + +1. **No controller is load-bearing at smoke scale.** The V7 production-readiness question — "can the policy stand without these safety-nets?" — answers *yes* at smoke scale. Every controller fires in < 50 % of epochs. The smoke-scale verdict is favourable but weak: several controllers cannot fire by construction here (see below). + +2. **Three controllers are structurally inert at smoke scale**, not diagnostically inert: + - **C1 (anti_lr)** is gated on `epoch >= 5` plus Sharpe outside ±threshold, and the fire *detector* reads the base scheduler which is `Constant` at smoke — double inertia. 0 / 60 fires is compatible with either "controller never triggers" or "detector never sees it" and the current wiring cannot distinguish. + - **C3 (gamma)** is pinned to the floor of `[0.9, 0.995]` because the `−0.05·(1 − health)` correction with smoke-scale `health ≈ 0.5` forces the effective value below 0.9 and the clamp absorbs the scheduled base's ticks. + - **C5 (cql_alpha)** gates on `(1 − regime_stability)` which is uninformative at smoke scale (ISV[11] degenerate). + C3 and C5 both hinge on the ISV `regime_stability` + `health` signals being real. Track-1 found `health ≈ 0.5 ± noise` at smoke scale. Until L40S populates those signals with meaningful values, the controllers' smoke-scale fire rates *under-report* their true adaptive load. All three promote to "DIAGNOSTIC at smoke, PENDING at production" until re-measured. + +3. **C2 (tau) and C5 (cql_alpha) fire only at fold boundaries.** Both fire twice out of 60, both at `HEALTH_DIAG[0]` of folds 2 and 3. This looks like fold-boundary state discontinuity — the training step counter resets, the cosine-annealed tau jumps by more than 1e-6 relative to the last epoch of the previous fold, and the cumulative `controller_fire_counts` doesn't reset per fold. Not a controller intervention in the reactive sense; a bookkeeping artefact of the multi-fold test structure. **If** the fold-boundary artefact were removed, C2 and C5 fire rates would drop to 0 / 60 at smoke scale — strengthening the DIAGNOSTIC verdict. + +4. **Anti_lr fire-detection semantics are incorrect** (see C1 wiring surprise). Under any decaying scheduler this will report false positives (scheduler ticks misattributed as anti_lr interventions). The intervention-based rewrite in `ed4b30b49` correctly handled C4 and C6 but did not correct C1. Fix in Phase 2 Track 3 before the L40S re-run, otherwise the L40S C1 verdict will itself be garbage. + +5. **C4 (grad_clip) is the only controller with meaningful intra-fold fire activity.** 12 fires distributed across all three folds, trending up in fold 3. At the currently-deployed `gradient_clip_norm=1.0` default (per MEMORY.md 2026-04-09) the EMA converges tight enough that occasional gradient spikes clip above it. Whether those spikes *matter* for training outcome is the open question and the only ablation Track 3 needs to run before the final verdict. + +6. **Interaction redundancy:** the four gradient-path controllers (C1 anti_lr, C4 grad_clip, C7 scheduler) all ultimately shape the effective step size. C3 (gamma) and C2 (tau) both scale future-return propagation but on different axes (discount vs target drift). C5 (cql_alpha) is orthogonal (conservatism penalty). No obvious redundancy pair worth flagging — the controllers shape distinct knobs. + +--- + +## Proposed Phase 2 changes (Track 3) + +Ordered by cost / evidence-quality payoff: + +1. **Fix C1 fire-detection wiring** (cheap; enables honest L40S measurement). Change `training_loop.rs:2460` to snapshot the post-anti_lr LR, OR detect anti_lr via the `anti_mult != 1.0` branch directly (`training_loop.rs:2936–2942`). Latter is unambiguous and matches the V7 intervention semantic; former risks re-introducing scheduler-drift false positives. Recommend the direct-branch approach. No behaviour change, only detection correctness. + +2. **Run C4 grad_clip ablation** (Track 3 Step 2 per plan). Disable adaptive clip (fixed `1e6` threshold) and re-run controller_activity + a short multi_fold. This is the only controller whose verdict requires data beyond the firing rate we already have. ~25 min per ablation on L40S. + +3. **Re-run Track 3 on L40S.** Multiple smoke-scale "DIAGNOSTIC-pending-scale" verdicts (C2, C3, C5) need production-scale HEALTH_DIAG to settle — in particular, `health` and `regime_stability` must not sit at 0.5 fallback values. + +4. **(Stretch) Fold-boundary counter reset.** If we want Track 3 verdicts to reflect a single clean 20-epoch run, reset `controller_fire_counts` and `prev_controller_values` at fold boundaries. Current cumulative behaviour spreads fold-boundary transients across the whole test, inflating `tau` and `cql_alpha` from 0/20 to 2/60. Optional; doesn't affect the qualitative verdicts. + +5. **(Follow-up, not Phase 2)** If the Track 1 H4 magnitude-gradient fix lands and training *does* produce real `health` swings, revisit C3 (gamma floor) and C5 (cql_alpha regime gate). Both currently look inert only because their input signals are degenerate; they may become materially active post-H4. + +--- + +## Limitations + +- **Smoke scale ≠ production scale.** `dqn-smoketest` profile, 176k-bar fxcache, batch=64, 5000-bar slice per iteration, RTX 3050 Ti. L40S production is batch=16384 / buffer=500k, with mature `health` and `regime_stability` signals. Controllers C2/C3/C5 all hinge on those signals — their smoke-scale fire rates are lower bounds, not representative. +- **No ablation data** for C4. The 20 % fire rate sits in the explicit ablation-required band in the spec decision matrix; until we run the grad-clip-off variant we cannot distinguish CANDIDATE FOR DELETE from LOAD-BEARING. +- **C1 fire-detection is wired incorrectly** for non-Constant LR decay (see C1 wiring surprise). The 0 / 60 reading at smoke scale happens to be correct (anti_lr never intervenes) but for the wrong reason (the detector would not see it even if it did, unless decay is Constant). +- **C6 and C7 are excluded from "load-bearing" by definition** (pure schedules). Their 0-fire readings in `[CTRL_FIRE]` are uninformative. The correct audit for these is "is the curriculum shape right?" — a Track-2 / hyperparameter question. +- **Cumulative counters, not per-fold.** Fire rates are reported over 60 epochs (3 folds × 20 epochs). Per-fold rates would be more noise-sensitive but diagnostic of fold-boundary artefacts. Left as an optional follow-up. + +--- + +## Next Track 3 steps + +- **Re-run on L40S** with the C1 fire-detection wiring fix applied, on the full Phase 0 baseline-capture branch, 20 epochs × 2 folds as per plan Task 1.1 Step 1. Promote each preliminary verdict to final. +- **Run the C4 grad-clip ablation** (Track 3 Step 2) and settle its CANDIDATE-FOR-DELETE verdict. +- **Proceed with Tracks 2 (Reward) and 4 (Exploration)** in parallel — all three tracks feed off the same HEALTH_DIAG run.