docs(policy-quality): Track 1 magnitude triage (Phase 1) — H4 + H10 CONFIRMED
Preliminary triage of spec §5.1 hypotheses H1–H10 using the Phase 0 baseline capture on RTX 3050 Ti. L40S validation pending per plan. Verdicts: H1 PENDING (needs forced-exploration instrumentation not yet wired) H2 REJECTED var_scale=0.96 across 19/20 epochs; Var[Q] inactive at smoke scale H3 INCONCLUSIVE kelly degenerate (insufficient win/loss counts at smoke scale) H4 CONFIRMED grad_ratio_mag_dir=0.0000 across 20/20 epochs (threshold <0.1) H5 REJECTED ent_mag stays ≥0.98 throughout; no bootstrap collapse H6 REJECTED Full fire rate (0) is lower than Quarter fire rate, not higher H7 REJECTED vsn and sigma symmetric between mag and dir branches H8 REJECTED target-net drift equal (mag=dir=0.001) H9 PENDING (same instrumentation gap as H1) H10 CONFIRMED training ent_mag=0.98, eval F_Quarter=100% Synthesis: H4 is the root cause. Magnitude branch receives ~0 gradient → weights stay near init → three magnitude Q-values near-identical → argmax picks bin 0 (Quarter) on ties → H10 manifests at eval time. H2, H5, H7, H8 all ruled out as contributors. Proposed Phase 2 priority: fix H4 (gradient-flow path into magnitude head — likely per-component advantage weighting or direction-conditioning of w_b1fc) + H10 (Q-margin argmax + stochastic eval rollouts as safety net). Phase 1 next: validate preliminary verdicts on L40S, instrument per-component gradient decomposition for magnitude, proceed with Tracks 2/3/4 in parallel.
This commit is contained in:
jgrusewski
@@ -0,0 +1,239 @@
|
||||
# Track 1 — Magnitude Collapse Triage
|
||||
|
||||
**Phase:** Phase 1 — Track 1 (Magnitude investigation)
|
||||
**Captured against commit:** `8ab368de5` (tag: `policy-quality-baseline`)
|
||||
**Hardware:** RTX 3050 Ti Laptop GPU (4 GB) — **preliminary smoke-scale verdicts**; L40S validation pending per plan §Task 1.1
|
||||
**Data source:** `magnitude_distribution.rs` smoke run (20 epochs, 176k-bar fxcache, `dqn-smoketest` profile) from the Phase 0 baseline capture.
|
||||
**Log ref:** `/tmp/foxhunt_smoke/magnitude_distribution.log` (local; not committed).
|
||||
|
||||
---
|
||||
|
||||
## Executive summary
|
||||
|
||||
Preliminary triage against spec §5.1 hypotheses. At smoke scale:
|
||||
|
||||
- **2 CONFIRMED** — H4 (magnitude-head gradient starvation), H10 (argmax tie-breaking at eval)
|
||||
- **5 REJECTED** — H2, H5, H6 (preliminary), H7, H8
|
||||
- **2 PENDING** — H1, H9 (both require the forced-exploration + per-magnitude-return instrumentation not yet in-tree)
|
||||
- **1 INCONCLUSIVE** — H3 (Kelly degenerate at smoke scale; needs a run long enough for meaningful win/loss counts)
|
||||
|
||||
**Most load-bearing finding**: H4 alone would explain the entire phenomenon — if the magnitude branch receives near-zero gradient for 20/20 epochs, it stays at initialization and cannot learn to prefer any magnitude. H10 (eval-mode argmax locks to the first bin on near-ties) compounds the observable symptom at evaluation time.
|
||||
|
||||
---
|
||||
|
||||
## Per-hypothesis verdicts
|
||||
|
||||
### H1 — Reward gradient asymmetry (Full-position variance > Quarter)
|
||||
|
||||
**Signal:** forced-exploration epoch (ε=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes, log realized step-return per (state, magnitude).
|
||||
|
||||
**Threshold:** `mean(R_full) / std(R_full) ≤ 0.5 × mean(R_quarter) / std(R_quarter)` → policy correctly prefers Quarter under Sharpe-per-trade logic.
|
||||
|
||||
**Observed:** not measured. The forced-exploration counterfactual experiment requires per-(state, magnitude) return logging that isn't yet instrumented — the kernel writes one realized step-return per sample, not all three magnitudes.
|
||||
|
||||
**Verdict:** **PENDING**
|
||||
|
||||
**To resolve:** Task 1.1 Step 2 of the plan (edit `training_loop.rs` to force `epsilon = 1.0` at epoch 10) + add a counterfactual-return logging hook that evaluates all three magnitudes for each of N sampled states. Not a one-liner — needs an additional per-(state, magnitude) buffer on the collector side.
|
||||
|
||||
---
|
||||
|
||||
### H2 — Var[Q] conviction-sizer over-shrinking
|
||||
|
||||
**Signal:** HEALTH_DIAG `mag [var_scale=…]`
|
||||
|
||||
**Threshold:** `var_scale_mean < 0.3` across ≥ 10 / 20 epochs.
|
||||
|
||||
**Observed trajectory (20 epochs)**:
|
||||
```
|
||||
0.314, 0.958, 0.959, 0.961, 0.965, 0.967, 0.968, 0.968, 0.969, 0.969,
|
||||
0.968, 0.968, 0.967, 0.967, 0.966, 0.966, 0.966, 0.960, 0.960, 0.961
|
||||
```
|
||||
Only epoch 0 is near-threshold (0.314); all 19 other epochs are ≥ 0.958. Var[Q] conviction scaling is essentially inactive at smoke scale (Var[Q] disabled for most samples because q_variance estimator needs warmup).
|
||||
|
||||
**Verdict:** **REJECTED** (1 / 20 epochs below threshold vs required 10 / 20).
|
||||
|
||||
**Caveat:** At production scale with warm Var[Q], this may tighten. Re-run at L40S before permanent REJECTED.
|
||||
|
||||
---
|
||||
|
||||
### H3 — Kelly as hard multiplier for marginal edges
|
||||
|
||||
**Signal:** HEALTH_DIAG `mag [kelly_f=… avg_win_ratio=…]`
|
||||
|
||||
**Threshold:** `kelly_f_mean ≤ 0.25` AND `avg_win / avg_loss < 1.3`.
|
||||
|
||||
**Observed trajectory**:
|
||||
- `kelly_f = 0.000` every epoch (20/20)
|
||||
- `avg_win_ratio = 0.000 to 0.003` every epoch
|
||||
|
||||
**Verdict:** **INCONCLUSIVE**
|
||||
|
||||
The smoke-scale data has degenerate win/loss statistics — `trade_stats_history.last()` returns a fold with insufficient win OR loss counts, so `kelly_f_mean` computation short-circuits to 0. The accessor is correct; the input is pathological at smoke scale. At production scale (50+ epochs, higher trade volume), kelly_f and avg_win_ratio should carry meaningful values.
|
||||
|
||||
**To resolve:** re-measure on L40S with 50-epoch training. If kelly_f_mean stays ≤ 0.25 with avg_win/avg_loss < 1.3 on real data, the hypothesis becomes CONFIRMED and the fix is to replace the Kelly hard-multiplier with a soft gate.
|
||||
|
||||
---
|
||||
|
||||
### H4 — Magnitude-head gradient starvation
|
||||
|
||||
**Signal:** HEALTH_DIAG `mag [grad_ratio_mag_dir=…]` — `‖∇ W[mag_head]‖ / ‖∇ W[dir_head]‖` per epoch.
|
||||
|
||||
**Threshold:** `grad_ratio_mag_dir < 0.1` across ≥ 15 / 20 epochs.
|
||||
|
||||
**Observed trajectory (20 epochs)**: `0.0000` on every single epoch (4-decimal precision).
|
||||
|
||||
Cross-check: the accessor returns `mag / dir` only when `dir > 1e-9`; else returns `0.0`. The `grad_stable` component of `learning_health.components` is `0.00` for most epochs (suggesting direction gradient is changing meaningfully) — so `dir_grad_norm > 1e-9` and the division is live, meaning `mag_grad_norm` is legitimately < `1e-4 * dir_grad_norm`.
|
||||
|
||||
**Verdict:** **CONFIRMED** (20 / 20 epochs below threshold vs required 15 / 20).
|
||||
|
||||
**Evidence strength:** STRONG. Magnitude branch receives gradient three-to-four orders of magnitude smaller than direction across the entire run. It cannot learn to distinguish Quarter / Half / Full if it never receives meaningful gradient signal. This alone would produce the observed phenomenon.
|
||||
|
||||
**Proposed fix** (per spec §5.1): rebalance 4-branch advantage weights OR adjust initial weight scale for the magnitude head. Investigating which branch of loss propagation skips magnitude is Phase 2 implementation work.
|
||||
|
||||
**Follow-up investigation needed**:
|
||||
- Which loss component (IQN / CQL / C51 / Ens) propagates gradient to magnitude head? Per-component grad norms would pin this down.
|
||||
- Is the direction-conditioned `w_b1fc` layout (magnitude FC depends on direction sliced input `[AH, SH2+3]`) blocking gradient flow? Direction-conditioning may inadvertently mask magnitude updates when direction is one-hot-dominant (Short or Long).
|
||||
|
||||
---
|
||||
|
||||
### H5 — Bootstrap bias (initial epsilon decayed too fast)
|
||||
|
||||
**Signal:** action entropy vs epsilon schedule.
|
||||
|
||||
**Threshold:** entropy < 0.3 before epoch 5 AND epsilon below 0.1.
|
||||
|
||||
**Observed**: `ent_mag` trajectory over 20 epochs: `0.98, 0.98, 0.98, 0.98, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.99, 0.98, 0.99, 0.98`. Magnitude entropy stays ≥ 0.98 throughout (near-uniform distribution across bins). Direction entropy (`ent_dir`) stays ≥ 0.98 as well.
|
||||
|
||||
**Verdict:** **REJECTED**
|
||||
|
||||
Entropy is *high*, not low. The policy is not stuck in a low-entropy-early pathology. If anything, the opposite is true — entropy is high throughout, suggesting NoisyNets and/or epsilon are well-mixed in training mode. The eval-time collapse (H10) is the relevant signal, not a training-time bootstrap.
|
||||
|
||||
---
|
||||
|
||||
### H6 — Regime-adaptive trailing stop closes Full positions early
|
||||
|
||||
**Signal:** HEALTH_DIAG `trail [fire_q=… fire_h=… fire_f=…]` per-magnitude trail firing rate.
|
||||
|
||||
**Threshold:** Full fire rate > 3× Quarter rate AND hold_time_at_exit on Full < 0.5× Quarter.
|
||||
|
||||
**Observed**: across the magnitude_distribution run, trail fire rates fluctuate between:
|
||||
- `fire_q = 0.12 – 0.66` per epoch (healthy Quarter trail activity)
|
||||
- `fire_h = 0.000` every epoch (Half positions too rare to generate fires)
|
||||
- `fire_f = 0.000` every epoch (Full positions too rare)
|
||||
|
||||
Full fire rate is *lower* than Quarter, not higher. Hypothesis direction contradicts the data.
|
||||
|
||||
**Verdict:** **REJECTED** (preliminary — very low Full-position sample size; re-measure at L40S scale for a strict rejection).
|
||||
|
||||
**Caveat:** `fire_f = 0` may simply mean "no Full-position trades ever entered that could trail-stop", which is itself a consequence of the magnitude collapse rather than a rebuttal of the trail-stop hypothesis. At production scale where Full positions actually enter (if H4/H10 are fixed), re-measure.
|
||||
|
||||
---
|
||||
|
||||
### H7 — VSN / NoisyNets noise masks magnitude more than direction
|
||||
|
||||
**Signal:** HEALTH_DIAG `noisy [vsn_mag=… vsn_dir=… sigma_mag=… sigma_dir=…]`.
|
||||
|
||||
**Threshold:** `vsn_mag < 0.5 × vsn_dir` OR `sigma_mag > 2 × sigma_dir`.
|
||||
|
||||
**Observed (stable across 20 epochs)**:
|
||||
- `vsn_mag = 0.010`, `vsn_dir = 0.010` — ratio 1.0 (not < 0.5)
|
||||
- `sigma_mag = 0.0320`, `sigma_dir = 0.0320` — ratio 1.0 (not > 2)
|
||||
|
||||
**Verdict:** **REJECTED**
|
||||
|
||||
Both VSN projection-weight magnitudes and NoisyNets σ values are symmetric across magnitude and direction branches. No asymmetric noise treatment is present.
|
||||
|
||||
**Note on `vsn_mag`**: the accessor is a projection-weight surrogate (Task 0.6 partial — live VSN mask isn't materialized), so this REJECT holds on the proxy metric. If the live mask ever gets instrumented, H7 becomes worth re-testing.
|
||||
|
||||
---
|
||||
|
||||
### H8 — Target-network tau too slow for magnitude propagation
|
||||
|
||||
**Signal:** HEALTH_DIAG `noisy [drift_mag=… drift_dir=…]` — cumulative target/online drift per branch.
|
||||
|
||||
**Threshold:** `drift_mag < 0.1 × drift_dir` cumulatively.
|
||||
|
||||
**Observed (stable across 20 epochs)**: `drift_mag = 0.001`, `drift_dir = 0.001`. Ratio 1.0.
|
||||
|
||||
**Verdict:** **REJECTED**
|
||||
|
||||
Target-network drift is identical between magnitude and direction branches. This rules out "magnitude bins are sampled too rarely for target propagation to converge".
|
||||
|
||||
**Note:** drift = 0.001 in both branches is quite low. Combined with H4 (mag gradient ~0), the magnitude branch's target AND online weights both stay near init. A stronger reading: H4 is upstream of the target-drift symmetry — target drift equals online drift equals zero because nothing is learning. H8 is correctly REJECTED as the *proximate cause*, but the rejection says nothing about whether tau is appropriate — just that tau isn't preferentially slowing magnitude.
|
||||
|
||||
---
|
||||
|
||||
### H9 — Data favors Quarter (not a bug — Quarter IS optimal)
|
||||
|
||||
**Signal:** per-magnitude Sharpe during forced-exploration epoch (requires H1 experiment).
|
||||
|
||||
**Threshold:** all three magnitudes show statistically similar Sharpe → Quarter-collapse is correct behavior.
|
||||
|
||||
**Observed:** not measured (same instrumentation gap as H1).
|
||||
|
||||
**Verdict:** **PENDING**
|
||||
|
||||
**To resolve:** the forced-exploration experiment required for H1 produces the data for H9 as a by-product. Defer both until instrumented.
|
||||
|
||||
**Side note:** H4 being CONFIRMED independent of H9 already means "the magnitude branch *isn't allowed to choose*", so even if H9 is also true, the pathway isn't through learning. H9 would be a "why remove magnitude" argument, not a "why the policy collapsed" explanation.
|
||||
|
||||
---
|
||||
|
||||
### H10 — Entropy regularization + argmax tie-breaking at eval
|
||||
|
||||
**Signal:** training-mode action distribution vs eval-mode action distribution.
|
||||
|
||||
**Threshold:** training `ent_mag > 0.8` (near-uniform) AND eval `F_Quarter > 90%`.
|
||||
|
||||
**Observed**:
|
||||
- Training `ent_mag = 0.86 – 0.99` across all 20 epochs (always ≥ 0.8)
|
||||
- Eval `eval_dist [eq=1.000 eh=0.000 ef=0.000]` every epoch (F_Quarter = 100%, well above 90%)
|
||||
|
||||
**Verdict:** **CONFIRMED** (strong).
|
||||
|
||||
**Evidence strength:** STRONG. The diagnostic produces a perfect signal — training distribution has near-maximum entropy (model outputs uniform), but eval argmax collapses 100% to bin 0 (Quarter). This is exactly the shape predicted by "uniform Q-values + stable argmax tie-break picks first bin".
|
||||
|
||||
**Combined with H4**: since H4 CONFIRMED means the magnitude branch hasn't learned to distinguish anything, its output IS near-uniform (three near-equal Q-values). Argmax over near-equal values is numerically unstable; Rust's `.iter().max_by()` returns the first-seen index on ties (Quarter = bin 0). H10 is the *observable consequence* of H4's underlying cause.
|
||||
|
||||
**Proposed fix** (per spec §5.1): break argmax ties by Q-value margin (no bin-index bias) OR match training / eval distributions via stochastic eval rollouts. The root-cause fix is H4; H10 is a safety-net patch to prevent accidentally relying on argmax tie-breaking order.
|
||||
|
||||
---
|
||||
|
||||
## Cross-hypothesis synthesis
|
||||
|
||||
1. **H4 is the root cause** at smoke scale. Magnitude branch gets effectively zero gradient → weights stay near init → all three magnitude Q-values are near-identical → argmax picks bin 0 (Quarter) on ties → H10 manifests at eval time.
|
||||
|
||||
2. **H2, H5, H7, H8 are ruled out** as contributors: Var[Q], bootstrap, VSN, and target-drift are all symmetric between mag and dir, or inactive at smoke scale.
|
||||
|
||||
3. **H3 needs production-scale data** (Kelly formula degenerates on low-volume trades). If H3 also CONFIRMS at L40S, a Kelly soft-gate is a compound fix on top of H4.
|
||||
|
||||
4. **H1 and H9 need new instrumentation** (forced-exploration counterfactual). Both could land in a Phase-0-bis micro-task if judged worth the scope — but the H4 fix alone should open up enough magnitude variation for Phase 2 to revisit H1/H9 with real distribution data.
|
||||
|
||||
5. **H6 needs real Full-position trades** before it can be fairly tested. Currently trivially-REJECTED by the data, but the data is itself a product of the collapse.
|
||||
|
||||
## Proposed Phase 2 fix priority order
|
||||
|
||||
1. **Primary (H4)**: diagnose gradient-flow path into the magnitude head. Likely candidates:
|
||||
- Per-component advantage weighting in the 4-branch head (direction branch may consume most of the per-batch gradient budget)
|
||||
- Direction-conditioning of `w_b1fc` ([AH, SH2+3]) — when direction is one-hot-dominant, the `+3` input slice is effectively zeroed for non-active directions
|
||||
- Initial weight scale for magnitude head (Xavier may be too small relative to direction head)
|
||||
2. **Secondary (H10)**: replace argmax tie-breaking at eval with Q-margin-aware selection OR add stochastic eval rollouts. Low cost, high safety-net value — prevents the symptom from reappearing even if H4 regresses.
|
||||
3. **Post-fix validation**: re-run baseline smoke + multi_fold with H4 + H10 patches applied. Expect:
|
||||
- `grad_ratio_mag_dir` > 0.1 across most epochs
|
||||
- Eval-mode `eq / eh / ef` converges toward training distribution (not 1/0/0)
|
||||
- `q_full`, `q_half`, `q_quarter` spread meaningfully (currently clustered at ~-0.05)
|
||||
|
||||
---
|
||||
|
||||
## Limitations of this preliminary triage
|
||||
|
||||
- **Smoke scale ≠ production scale.** `dqn-smoketest` uses batch=64, buffer=256, 5000-bar data slice. L40S production is batch=16384, buffer=500k, 176k bars. Some hypotheses may flip direction at scale (H2, H3, H6).
|
||||
- **4 GB VRAM constrains what we can test locally.** No production-scale ablations.
|
||||
- **Per-component gradient decomposition missing.** H4 is CONFIRMED but we don't yet know *which* loss component (IQN / CQL / C51 / Ens) sends zero gradient to magnitude. Phase 2 needs that breakdown before choosing a fix.
|
||||
- **H1 / H9 not measured.** The forced-exploration counterfactual is the one measurement not yet wired; both would give useful extra evidence if instrumented.
|
||||
|
||||
## Next Phase 1 steps
|
||||
|
||||
- Re-run this triage on L40S (20 epochs × 2 folds, per plan Task 1.1 Step 1) to promote each preliminary verdict to final.
|
||||
- Instrument per-component gradient norms for the magnitude head (IQN / CQL / C51 / Ens split) to localize H4.
|
||||
- Proceed with Tracks 2 (Reward V7 audit), 3 (Controllers), 4 (Exploration) in parallel — they reuse the same baseline run's HEALTH_DIAG output.
|
||||
Reference in New Issue
Block a user