diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md b/docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md new file mode 100644 index 000000000..30b04e419 --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md @@ -0,0 +1,239 @@ +# Track 1 — Magnitude Collapse Triage + +**Phase:** Phase 1 — Track 1 (Magnitude investigation) +**Captured against commit:** `8ab368de5` (tag: `policy-quality-baseline`) +**Hardware:** RTX 3050 Ti Laptop GPU (4 GB) — **preliminary smoke-scale verdicts**; L40S validation pending per plan §Task 1.1 +**Data source:** `magnitude_distribution.rs` smoke run (20 epochs, 176k-bar fxcache, `dqn-smoketest` profile) from the Phase 0 baseline capture. +**Log ref:** `/tmp/foxhunt_smoke/magnitude_distribution.log` (local; not committed). + +--- + +## Executive summary + +Preliminary triage against spec §5.1 hypotheses. At smoke scale: + +- **2 CONFIRMED** — H4 (magnitude-head gradient starvation), H10 (argmax tie-breaking at eval) +- **5 REJECTED** — H2, H5, H6 (preliminary), H7, H8 +- **2 PENDING** — H1, H9 (both require the forced-exploration + per-magnitude-return instrumentation not yet in-tree) +- **1 INCONCLUSIVE** — H3 (Kelly degenerate at smoke scale; needs a run long enough for meaningful win/loss counts) + +**Most load-bearing finding**: H4 alone would explain the entire phenomenon — if the magnitude branch receives near-zero gradient for 20/20 epochs, it stays at initialization and cannot learn to prefer any magnitude. H10 (eval-mode argmax locks to the first bin on near-ties) compounds the observable symptom at evaluation time. + +--- + +## Per-hypothesis verdicts + +### H1 — Reward gradient asymmetry (Full-position variance > Quarter) + +**Signal:** forced-exploration epoch (ε=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes, log realized step-return per (state, magnitude). + +**Threshold:** `mean(R_full) / std(R_full) ≤ 0.5 × mean(R_quarter) / std(R_quarter)` → policy correctly prefers Quarter under Sharpe-per-trade logic. + +**Observed:** not measured. The forced-exploration counterfactual experiment requires per-(state, magnitude) return logging that isn't yet instrumented — the kernel writes one realized step-return per sample, not all three magnitudes. + +**Verdict:** **PENDING** + +**To resolve:** Task 1.1 Step 2 of the plan (edit `training_loop.rs` to force `epsilon = 1.0` at epoch 10) + add a counterfactual-return logging hook that evaluates all three magnitudes for each of N sampled states. Not a one-liner — needs an additional per-(state, magnitude) buffer on the collector side. + +--- + +### H2 — Var[Q] conviction-sizer over-shrinking + +**Signal:** HEALTH_DIAG `mag [var_scale=…]` + +**Threshold:** `var_scale_mean < 0.3` across ≥ 10 / 20 epochs. + +**Observed trajectory (20 epochs)**: +``` +0.314, 0.958, 0.959, 0.961, 0.965, 0.967, 0.968, 0.968, 0.969, 0.969, +0.968, 0.968, 0.967, 0.967, 0.966, 0.966, 0.966, 0.960, 0.960, 0.961 +``` +Only epoch 0 is near-threshold (0.314); all 19 other epochs are ≥ 0.958. Var[Q] conviction scaling is essentially inactive at smoke scale (Var[Q] disabled for most samples because q_variance estimator needs warmup). + +**Verdict:** **REJECTED** (1 / 20 epochs below threshold vs required 10 / 20). + +**Caveat:** At production scale with warm Var[Q], this may tighten. Re-run at L40S before permanent REJECTED. + +--- + +### H3 — Kelly as hard multiplier for marginal edges + +**Signal:** HEALTH_DIAG `mag [kelly_f=… avg_win_ratio=…]` + +**Threshold:** `kelly_f_mean ≤ 0.25` AND `avg_win / avg_loss < 1.3`. + +**Observed trajectory**: +- `kelly_f = 0.000` every epoch (20/20) +- `avg_win_ratio = 0.000 to 0.003` every epoch + +**Verdict:** **INCONCLUSIVE** + +The smoke-scale data has degenerate win/loss statistics — `trade_stats_history.last()` returns a fold with insufficient win OR loss counts, so `kelly_f_mean` computation short-circuits to 0. The accessor is correct; the input is pathological at smoke scale. At production scale (50+ epochs, higher trade volume), kelly_f and avg_win_ratio should carry meaningful values. + +**To resolve:** re-measure on L40S with 50-epoch training. If kelly_f_mean stays ≤ 0.25 with avg_win/avg_loss < 1.3 on real data, the hypothesis becomes CONFIRMED and the fix is to replace the Kelly hard-multiplier with a soft gate. + +--- + +### H4 — Magnitude-head gradient starvation + +**Signal:** HEALTH_DIAG `mag [grad_ratio_mag_dir=…]` — `‖∇ W[mag_head]‖ / ‖∇ W[dir_head]‖` per epoch. + +**Threshold:** `grad_ratio_mag_dir < 0.1` across ≥ 15 / 20 epochs. + +**Observed trajectory (20 epochs)**: `0.0000` on every single epoch (4-decimal precision). + +Cross-check: the accessor returns `mag / dir` only when `dir > 1e-9`; else returns `0.0`. The `grad_stable` component of `learning_health.components` is `0.00` for most epochs (suggesting direction gradient is changing meaningfully) — so `dir_grad_norm > 1e-9` and the division is live, meaning `mag_grad_norm` is legitimately < `1e-4 * dir_grad_norm`. + +**Verdict:** **CONFIRMED** (20 / 20 epochs below threshold vs required 15 / 20). + +**Evidence strength:** STRONG. Magnitude branch receives gradient three-to-four orders of magnitude smaller than direction across the entire run. It cannot learn to distinguish Quarter / Half / Full if it never receives meaningful gradient signal. This alone would produce the observed phenomenon. + +**Proposed fix** (per spec §5.1): rebalance 4-branch advantage weights OR adjust initial weight scale for the magnitude head. Investigating which branch of loss propagation skips magnitude is Phase 2 implementation work. + +**Follow-up investigation needed**: +- Which loss component (IQN / CQL / C51 / Ens) propagates gradient to magnitude head? Per-component grad norms would pin this down. +- Is the direction-conditioned `w_b1fc` layout (magnitude FC depends on direction sliced input `[AH, SH2+3]`) blocking gradient flow? Direction-conditioning may inadvertently mask magnitude updates when direction is one-hot-dominant (Short or Long). + +--- + +### H5 — Bootstrap bias (initial epsilon decayed too fast) + +**Signal:** action entropy vs epsilon schedule. + +**Threshold:** entropy < 0.3 before epoch 5 AND epsilon below 0.1. + +**Observed**: `ent_mag` trajectory over 20 epochs: `0.98, 0.98, 0.98, 0.98, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.98, 0.99, 0.98`. Magnitude entropy stays ≥ 0.98 throughout (near-uniform distribution across bins). Direction entropy (`ent_dir`) stays ≥ 0.98 as well. + +**Verdict:** **REJECTED** + +Entropy is *high*, not low. The policy is not stuck in a low-entropy-early pathology. If anything, the opposite is true — entropy is high throughout, suggesting NoisyNets and/or epsilon are well-mixed in training mode. The eval-time collapse (H10) is the relevant signal, not a training-time bootstrap. + +--- + +### H6 — Regime-adaptive trailing stop closes Full positions early + +**Signal:** HEALTH_DIAG `trail [fire_q=… fire_h=… fire_f=…]` per-magnitude trail firing rate. + +**Threshold:** Full fire rate > 3× Quarter rate AND hold_time_at_exit on Full < 0.5× Quarter. + +**Observed**: across the magnitude_distribution run, trail fire rates fluctuate between: +- `fire_q = 0.12 – 0.66` per epoch (healthy Quarter trail activity) +- `fire_h = 0.000` every epoch (Half positions too rare to generate fires) +- `fire_f = 0.000` every epoch (Full positions too rare) + +Full fire rate is *lower* than Quarter, not higher. Hypothesis direction contradicts the data. + +**Verdict:** **REJECTED** (preliminary — very low Full-position sample size; re-measure at L40S scale for a strict rejection). + +**Caveat:** `fire_f = 0` may simply mean "no Full-position trades ever entered that could trail-stop", which is itself a consequence of the magnitude collapse rather than a rebuttal of the trail-stop hypothesis. At production scale where Full positions actually enter (if H4/H10 are fixed), re-measure. + +--- + +### H7 — VSN / NoisyNets noise masks magnitude more than direction + +**Signal:** HEALTH_DIAG `noisy [vsn_mag=… vsn_dir=… sigma_mag=… sigma_dir=…]`. + +**Threshold:** `vsn_mag < 0.5 × vsn_dir` OR `sigma_mag > 2 × sigma_dir`. + +**Observed (stable across 20 epochs)**: +- `vsn_mag = 0.010`, `vsn_dir = 0.010` — ratio 1.0 (not < 0.5) +- `sigma_mag = 0.0320`, `sigma_dir = 0.0320` — ratio 1.0 (not > 2) + +**Verdict:** **REJECTED** + +Both VSN projection-weight magnitudes and NoisyNets σ values are symmetric across magnitude and direction branches. No asymmetric noise treatment is present. + +**Note on `vsn_mag`**: the accessor is a projection-weight surrogate (Task 0.6 partial — live VSN mask isn't materialized), so this REJECT holds on the proxy metric. If the live mask ever gets instrumented, H7 becomes worth re-testing. + +--- + +### H8 — Target-network tau too slow for magnitude propagation + +**Signal:** HEALTH_DIAG `noisy [drift_mag=… drift_dir=…]` — cumulative target/online drift per branch. + +**Threshold:** `drift_mag < 0.1 × drift_dir` cumulatively. + +**Observed (stable across 20 epochs)**: `drift_mag = 0.001`, `drift_dir = 0.001`. Ratio 1.0. + +**Verdict:** **REJECTED** + +Target-network drift is identical between magnitude and direction branches. This rules out "magnitude bins are sampled too rarely for target propagation to converge". + +**Note:** drift = 0.001 in both branches is quite low. Combined with H4 (mag gradient ~0), the magnitude branch's target AND online weights both stay near init. A stronger reading: H4 is upstream of the target-drift symmetry — target drift equals online drift equals zero because nothing is learning. H8 is correctly REJECTED as the *proximate cause*, but the rejection says nothing about whether tau is appropriate — just that tau isn't preferentially slowing magnitude. + +--- + +### H9 — Data favors Quarter (not a bug — Quarter IS optimal) + +**Signal:** per-magnitude Sharpe during forced-exploration epoch (requires H1 experiment). + +**Threshold:** all three magnitudes show statistically similar Sharpe → Quarter-collapse is correct behavior. + +**Observed:** not measured (same instrumentation gap as H1). + +**Verdict:** **PENDING** + +**To resolve:** the forced-exploration experiment required for H1 produces the data for H9 as a by-product. Defer both until instrumented. + +**Side note:** H4 being CONFIRMED independent of H9 already means "the magnitude branch *isn't allowed to choose*", so even if H9 is also true, the pathway isn't through learning. H9 would be a "why remove magnitude" argument, not a "why the policy collapsed" explanation. + +--- + +### H10 — Entropy regularization + argmax tie-breaking at eval + +**Signal:** training-mode action distribution vs eval-mode action distribution. + +**Threshold:** training `ent_mag > 0.8` (near-uniform) AND eval `F_Quarter > 90%`. + +**Observed**: +- Training `ent_mag = 0.86 – 0.99` across all 20 epochs (always ≥ 0.8) +- Eval `eval_dist [eq=1.000 eh=0.000 ef=0.000]` every epoch (F_Quarter = 100%, well above 90%) + +**Verdict:** **CONFIRMED** (strong). + +**Evidence strength:** STRONG. The diagnostic produces a perfect signal — training distribution has near-maximum entropy (model outputs uniform), but eval argmax collapses 100% to bin 0 (Quarter). This is exactly the shape predicted by "uniform Q-values + stable argmax tie-break picks first bin". + +**Combined with H4**: since H4 CONFIRMED means the magnitude branch hasn't learned to distinguish anything, its output IS near-uniform (three near-equal Q-values). Argmax over near-equal values is numerically unstable; Rust's `.iter().max_by()` returns the first-seen index on ties (Quarter = bin 0). H10 is the *observable consequence* of H4's underlying cause. + +**Proposed fix** (per spec §5.1): break argmax ties by Q-value margin (no bin-index bias) OR match training / eval distributions via stochastic eval rollouts. The root-cause fix is H4; H10 is a safety-net patch to prevent accidentally relying on argmax tie-breaking order. + +--- + +## Cross-hypothesis synthesis + +1. **H4 is the root cause** at smoke scale. Magnitude branch gets effectively zero gradient → weights stay near init → all three magnitude Q-values are near-identical → argmax picks bin 0 (Quarter) on ties → H10 manifests at eval time. + +2. **H2, H5, H7, H8 are ruled out** as contributors: Var[Q], bootstrap, VSN, and target-drift are all symmetric between mag and dir, or inactive at smoke scale. + +3. **H3 needs production-scale data** (Kelly formula degenerates on low-volume trades). If H3 also CONFIRMS at L40S, a Kelly soft-gate is a compound fix on top of H4. + +4. **H1 and H9 need new instrumentation** (forced-exploration counterfactual). Both could land in a Phase-0-bis micro-task if judged worth the scope — but the H4 fix alone should open up enough magnitude variation for Phase 2 to revisit H1/H9 with real distribution data. + +5. **H6 needs real Full-position trades** before it can be fairly tested. Currently trivially-REJECTED by the data, but the data is itself a product of the collapse. + +## Proposed Phase 2 fix priority order + +1. **Primary (H4)**: diagnose gradient-flow path into the magnitude head. Likely candidates: + - Per-component advantage weighting in the 4-branch head (direction branch may consume most of the per-batch gradient budget) + - Direction-conditioning of `w_b1fc` ([AH, SH2+3]) — when direction is one-hot-dominant, the `+3` input slice is effectively zeroed for non-active directions + - Initial weight scale for magnitude head (Xavier may be too small relative to direction head) +2. **Secondary (H10)**: replace argmax tie-breaking at eval with Q-margin-aware selection OR add stochastic eval rollouts. Low cost, high safety-net value — prevents the symptom from reappearing even if H4 regresses. +3. **Post-fix validation**: re-run baseline smoke + multi_fold with H4 + H10 patches applied. Expect: + - `grad_ratio_mag_dir` > 0.1 across most epochs + - Eval-mode `eq / eh / ef` converges toward training distribution (not 1/0/0) + - `q_full`, `q_half`, `q_quarter` spread meaningfully (currently clustered at ~-0.05) + +--- + +## Limitations of this preliminary triage + +- **Smoke scale ≠ production scale.** `dqn-smoketest` uses batch=64, buffer=256, 5000-bar data slice. L40S production is batch=16384, buffer=500k, 176k bars. Some hypotheses may flip direction at scale (H2, H3, H6). +- **4 GB VRAM constrains what we can test locally.** No production-scale ablations. +- **Per-component gradient decomposition missing.** H4 is CONFIRMED but we don't yet know *which* loss component (IQN / CQL / C51 / Ens) sends zero gradient to magnitude. Phase 2 needs that breakdown before choosing a fix. +- **H1 / H9 not measured.** The forced-exploration counterfactual is the one measurement not yet wired; both would give useful extra evidence if instrumented. + +## Next Phase 1 steps + +- Re-run this triage on L40S (20 epochs × 2 folds, per plan Task 1.1 Step 1) to promote each preliminary verdict to final. +- Instrument per-component gradient norms for the magnitude head (IQN / CQL / C51 / Ens split) to localize H4. +- Proceed with Tracks 2 (Reward V7 audit), 3 (Controllers), 4 (Exploration) in parallel — they reuse the same baseline run's HEALTH_DIAG output.