diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-design.md b/docs/superpowers/specs/2026-04-21-policy-quality-design.md index 3f22d5954..d70ba2f23 100644 --- a/docs/superpowers/specs/2026-04-21-policy-quality-design.md +++ b/docs/superpowers/specs/2026-04-21-policy-quality-design.md @@ -40,7 +40,7 @@ The spec closes when a single L40S validation run (6-fold × 50-epoch, the exist - **B-1 Multi-fold Sharpe:** Best val Sharpe (annualized) `> 10` on at least **5/6 folds** - **B-2 Multi-fold WinRate:** Val WinRate `> 55%` on at least **5/6 folds** -- **SN Surrogate-noise:** Random-action Sharpe `≤ 0.5×` trained-model Sharpe on same val data (`surrogate_noise_check.rs`) +- **SN Surrogate-noise:** Trained-model pooled Sharpe `>` 95th-percentile of N=30 random-action surrogate runs (`surrogate_noise_check.rs`). Uses pooled-across-6-folds val returns (~600 trades) so the comparison has statistical power. ### 2.2 Soft gates (ship with documented follow-up if missed) @@ -53,7 +53,11 @@ The spec closes when a single L40S validation run (6-fold × 50-epoch, the exist - **All mandatory + all soft pass** → merge to main, close spec clean. - **All mandatory pass, some soft miss** → merge to main, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. -- **Any mandatory miss** → `feat/policy-quality` stays unmerged. Up to 2 more iterations on the feature branch. If still failing, scope exceeds this spec — escalate to sub-project A v2. +- **Any mandatory miss** → `feat/policy-quality` stays unmerged. Up to 2 more iterations on the feature branch (total 3 Phase-3 validation runs). If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with: + - Summary of the 3 failed runs' evidence (which gates, by how much, which hypotheses correlate with failure) + - Explicit diagnosis: policy/reward/state bottleneck? architecture bottleneck? data bottleneck? + - Decision: continue on `feat/policy-quality` with new hypotheses, OR fork a new branch `feat/policy-quality-v2` with a different angle (e.g., reward redesign, architecture change, alternative algorithm PPO/SAC) + - Triage spec is a full brainstorming cycle (not a follow-up note) — restarts the skill flow. --- @@ -149,9 +153,11 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga **`surrogate_noise_check.rs`** — false-positive detector - Use the trained model from Phase 2 or Phase 3. -- On validation data, REPLACE model actions with uniformly random actions (same distribution shape as trained model). -- Run backtest with random actions. -- Assert: random-action Sharpe ≤ 0.5× trained-model Sharpe. If random actions produce comparable Sharpe, we have measurement/leakage bugs (look-ahead, data re-use, stale features) — not a real edge. +- **Pool all 6 validation folds** into a single val-return stream (~600 trades aggregated) — per-fold Sharpe is sample-size-noisy, so the per-fold comparison could be satisfied by luck. Pooled Sharpe has the statistical power needed for this gate. +- Generate **N=30 surrogate runs**: for each, REPLACE model actions with uniformly random actions (matched marginal action distribution to the trained model's, so the comparison isn't trivially won by action-frequency differences alone). Run backtest per surrogate, collect 30 random-action pooled Sharpes. +- Compute the 95th percentile of the surrogate Sharpe distribution. +- Assert: **trained-model pooled Sharpe > surrogate 95th-percentile Sharpe** (i.e., the trained model is outside the top 5% of what random actions produce by chance). +- Rationale for statistical gate: "0.5× ratio" in a single comparison is vulnerable to sample noise; a percentile-based test gives genuine statistical power with explicit false-positive control. - Critical production-readiness gate. Permanent artifact — guards against future measurement regressions. ### 4.2 HEALTH_DIAG extensions @@ -159,11 +165,21 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga Add to the existing HEALTH_DIAG log line in `training_loop.rs`. All forward-only, no backward gradient, cost is one scalar readback per epoch per field. Track 1 fields (magnitude diagnosis): + +Covering H1–H5 (common causes): - `q_mag_full`, `q_mag_half`, `q_mag_quarter` — mean Q-value per magnitude bin -- `var_scale_mean` — average Var[Q]-derived conviction shrink -- `kelly_f_mean` — average Kelly fraction applied -- `grad_ratio_mag_dir` — magnitude-head grad norm ÷ direction-head grad norm -- `action_dist_quarter`, `action_dist_half`, `action_dist_full` — epoch-level usage fractions +- `var_scale_mean` — average Var[Q]-derived conviction shrink (H2) +- `kelly_f_mean`, `avg_win_ratio` — Kelly fraction + `avg_win / avg_loss` ratio (H3) +- `grad_ratio_mag_dir` — magnitude-head grad norm ÷ direction-head grad norm (H4) +- `action_dist_quarter`, `action_dist_half`, `action_dist_full` — epoch-level usage fractions (H5 indirect) + +Covering H6–H10 (codebase-specific): +- `trail_fire_rate_{quarter, half, full}` — trailing-stop firing rate per magnitude bin (H6) +- `hold_time_at_exit_{quarter, half, full}` — mean bars held until exit, per magnitude (H6) +- `vsn_mask_{magnitude, direction}` — VSN mask mean per branch (H7) +- `noisy_sigma_{mag_head, dir_head}` — NoisyNets σ mean per branch's weights (H7) +- `target_drift_{mag_head, dir_head}` — cumulative ‖target − online‖ per branch (H8) +- `action_dist_eval_{quarter, half, full}` — magnitude usage during evaluation rollouts (distinct from training-mode `action_dist_*`) (H10) Track 2 fields (reward V7 audit): - `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return (see class-specific measurement in §4.1) @@ -412,7 +428,7 @@ Existing walk-forward config produces 6 folds on `test_data/futures-baseline`. | A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel | | **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log | | **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel | -| **Surrogate-noise (MANDATORY)** | Random-action Sharpe ≤ 0.5× trained-model Sharpe | `surrogate_noise_check.rs` smoke | +| **Surrogate-noise (MANDATORY)** | Trained-model pooled Sharpe > 95th-percentile of N=30 surrogate random-action runs | `surrogate_noise_check.rs` smoke | | Controller-activity (soft) | No controller fires in > 50% of epochs | HEALTH_DIAG `controller_fire_*` fields | Criterion B gates AND surrogate-noise check MUST pass for the spec to close successfully and merge to main (Phase 4). Criterion A gates are soft — if B and surrogate-noise pass but some A gate misses, the spec closes with a documented A follow-up. Controller-activity is soft — high firing rates go into a tech-debt follow-up note (sub-project B concern).