design(policy-quality): self-review revisions

Critical fixes after honest review:
  1. reward-audit measurement: split into 3 classes (additive / transform
     / sample-selector) — one-size-fits-all metric couldn't work for
     PopArt normalization or CF flip mirroring
  2. H1 detection: replace undefined "Sharpe/state" with concrete
     forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train,
     sample states × magnitudes, compare Sharpe-per-trade)
  3. H4 scope: clarify this is the per-action-branch gradient (direction
     vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens)
     budget. Fix sketch updated to target the correct knob.
  4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now
     MANDATORY; criterion A gates are soft. Prevents "partial success"
     hiding a policy-quality failure.
  5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean
     folds pooled with 1 fat one should not pass.

Minor:
  - Budget trimmed to 3.5–4 hrs L40S (was 6–10)
  - L40S pool made explicit as target
  - Phase 1 branches now pushed as wip/* for crash safety
  - Hyperopt-params mismatch handling documented

docs/superpowers/specs/2026-04-21-policy-quality-design.md

@@ -82,7 +82,15 @@ main timeline:
 - **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production.
 - **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence.
 
-**Budget estimate:** ~3–5 days elapsed. ~6–10 hours L40S GPU time total (≈1 hour per validation run, 2–3 iterations expected).
+**Budget estimate:** ~3–5 days elapsed.
+
+- L40S validation runs: ~1 hour each, budget 3 runs = **3 hrs** (one after Phase 2, up to 2 more to iterate if partial)
+- Local smoke runs (RTX 3050): ~3 min each, budget unlimited
+- Investigation HEALTH_DIAG runs during Phase 1 use the same Argo workflows, 20-epoch × 2-fold = ~12 min on L40S, budget 3 = ~30 min
+
+Total GPU: **~3.5–4 hrs L40S** (not 6–10 — earlier estimate was conservative).
+
+**Target GPU pool:** `ci-training-l40s` throughout. H100 only if L40S queue saturates (no expected architecture-specific differences).
 
 ---
 
@@ -100,9 +108,13 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga
 - HEALTH_DIAG readback: per-magnitude Q means, var_scale mean, kelly_f mean, per-branch grad ratio, action distribution per bin.
 
 **`reward_component_audit.rs`** — Track 2 gate
-- Train 20 epochs, record per-term `|contribution| / |total step_return|` for each of the 8 reward terms (R1–R8 enumerated below).
-- Assert: each ACTIVE term has `|contribution|/|total| ≥ 1%` in `≥ 50%` of epochs.
-- Terms below threshold are triaged for DELETE in Phase 2.
+- Train 20 epochs.
+- Reward terms fall into three measurement classes, each with its own detection:
+  - **Additive** (R1 step_return, R5 micro-reward, R6 loss-aversion, R7 segment-patience): record `|term| / |total reward|` per step, assert the active term's median ratio ≥ 1% across epochs.
+  - **Transform** (R2 PopArt): record `|post_popart − pre_popart|`; if < 10% of |pre_popart| typical scale, PopArt is no-op-equivalent and DELETE-triage.
+  - **Sample-selector** (R3 CF-flip, R4 trailing-stop): record activation rate (fraction of samples/episodes the term fires on) AND downstream effect (grad-norm delta on fired vs unfired batches). Sub-noise = <1% firing rate OR indistinguishable grad signal.
+- Terms below their class's threshold are triaged for DELETE in Phase 2.
+- Implementation detail: the test uses one training run with instrumentation, not 8 separate runs.
 
 **`multi_fold_convergence.rs`** — fast local variant of the full gate
 - 3 folds × 20 epochs (vs 6 × 50 on L40S). Runs ~5 min on RTX 3050.
@@ -132,7 +144,13 @@ Track 2 fields (reward V7 audit):
 
 ## 5. Phase 1 — Parallel investigation tracks
 
-No commits land during Phase 1. Findings live in uncommitted worktree or throwaway local branches.
+**Branch discipline during Phase 1:**
+
+- No commits land on `main`.
+- Each track uses a throwaway local branch (`wip/policy-quality-track1`, `wip/policy-quality-track2`) that IS pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable.
+- Track branches are never merged. At Phase 2, findings are re-applied by hand on a fresh `main` branch, and the throwaway branches are deleted.
+
+This gives us crash safety (branches pushed) + baseline cleanliness (nothing lands on main).
 
 ### 5.1 Track 1 — Magnitude collapse diagnosis
 
@@ -140,10 +158,10 @@ Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED**
 
 | # | Hypothesis | Detection signal | Detection threshold |
 |---|---|---|---|
-| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | Per-bin reward mean + variance from replay buffer | Full Sharpe/state < 0.5× Quarter Sharpe/state across ≥10/20 epochs |
+| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | During a forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | Ratio `mean(R_full) / std(R_full)` is ≤ 0.5× `mean(R_quarter) / std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. |
 | H2 | Var[Q] conviction-sizer over-shrinking — `effective_max_pos *= 1/(1 + sqrt(Var[Q]))` permanently collapses the target toward Quarter. | `var_scale_mean` field | `var_scale_mean < 0.3` across ≥10/20 epochs |
 | H3 | Kelly cap early-firing — Kelly stats (computed from Quarter-only history) cap at Quarter after 20+ trades. | `kelly_f_mean` field | `kelly_f_mean ≤ 0.25` across ≥10/20 epochs |
-| H4 | Per-branch gradient under-allocation — magnitude head gets too small a share of the gradient budget. | `grad_ratio_mag_dir` field | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs |
+| H4 | Magnitude-head gradient starvation — the per-action-branch gradient (*not* the per-loss-component IQN/CQL/C51/Ens split) to the magnitude head is much smaller than to the direction head. Magnitude weights stay near init. | `grad_ratio_mag_dir` field, measured as end-of-epoch average of per-step ‖∇ W[mag_head]‖ ÷ ‖∇ W[direction_head]‖ (pre-clip). | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs |
 | H5 | Bootstrap bias — initial random exploration anchored Quarter, epsilon decayed before the policy escaped. | Action entropy vs epsilon schedule | entropy < 0.3 before epoch 5 AND epsilon below 0.1 |
 
 **Fix sketch (if confirmed):**
@@ -151,7 +169,7 @@ Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED**
 - H1 → rebalance reward gradient per magnitude (advantage weighting by bin, or per-bin loss scale)
 - H2 → clamp `var_scale ≥ 0.5` OR remove entirely if redundant with Kelly
 - H3 → decouple Kelly from magnitude conditioning (use Kelly as position scaler, not magnitude quantizer)
-- H4 → reallocate per-branch gradient budget (currently IQN=60%, CQL=25%, C51=10%, Ens=5% per ADR; may need explicit magnitude-head share)
+- H4 → investigate why magnitude-head gradient is small. Candidates: magnitude-branch loss weight too low in the 4-branch advantage sum, magnitude head's `w_b1fc` dimensionality causing numerical suppression, VSN (variable selection network) masking magnitude features. Fix by rebalancing the 4-branch advantage weights OR adjusting initial weight scale for the magnitude head. (NOT the IQN/CQL/C51/Ens loss-component budget — that's orthogonal.)
 - H5 → extend exploration warmup for magnitude branch only
 
 ### 5.2 Track 2 — Reward V7 audit
@@ -244,11 +262,13 @@ Existing walk-forward config produces 6 folds on `test_data/futures-baseline`.
 
 | Gate | Pass condition | Source |
 |---|---|---|
-| A — Action-dist | `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds | HEALTH_DIAG `action_dist_*`, absence of LOW EXPOSURE DIVERSITY warnings |
-| A — WinRate | Val WinRate in [55%, 65%] across folds | Metrics kernel |
-| A — Trade count | ≥100 trades per val window (avg across 6 folds) | Metrics kernel |
-| B — Multi-fold Sharpe | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log |
-| B — Multi-fold WinRate | Val WinRate > 55% on ≥5/6 folds | Metrics kernel |
+| A — Action-dist (soft) | `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds | HEALTH_DIAG `action_dist_*`, absence of LOW EXPOSURE DIVERSITY warnings |
+| A — WinRate (soft) | Val WinRate ≥ 55% across folds (upper-bound fine) | Metrics kernel |
+| A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel |
+| **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log |
+| **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel |
+
+Criterion B gates MUST pass for the spec to close successfully. Criterion A gates are soft — if B passes but some A gate misses, the spec closes with a documented A follow-up.
 
 ### 7.3 Partial-success handling
 
@@ -278,6 +298,7 @@ These are the regression coverage for the invariants this spec establishes.
 - Hyperopt re-runs → separate concern; current defaults are used as baseline
 - Architectural changes beyond action space (no new network heads, no new layers)
 - State representation changes (OFI features, regime features — kept as-is)
+- Hyperopt-params file regeneration. If Phase 2 changes action-space layout (e.g., adds a 4th magnitude bin), the existing `hyperopt_params.json` may have stale branch sizes. Policy: detect mismatch at runtime, log warning, use CLI branch-size defaults. A proper hyperopt re-run is a separate concern (sub-project B includes it implicitly as part of multi-fold iteration).
 
 ---