From bee81b4967be4d1ede01cb8abe0d6d6113e26f8b Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 21 Apr 2026 20:29:13 +0200 Subject: [PATCH] design(policy-quality): self-review revisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical fixes after honest review: 1. reward-audit measurement: split into 3 classes (additive / transform / sample-selector) — one-size-fits-all metric couldn't work for PopArt normalization or CF flip mirroring 2. H1 detection: replace undefined "Sharpe/state" with concrete forced-exploration protocol (epsilon=1.0 for 1 epoch at mid-train, sample states × magnitudes, compare Sharpe-per-trade) 3. H4 scope: clarify this is the per-action-branch gradient (direction vs magnitude head), NOT the per-loss-component (IQN/CQL/C51/Ens) budget. Fix sketch updated to target the correct knob. 4. Validation gate: criterion B (multi-fold Sharpe + WinRate) is now MANDATORY; criterion A gates are soft. Prevents "partial success" hiding a policy-quality failure. 5. Trade-count gate: per-fold ≥100 on ≥5/6, not an average — 5 lean folds pooled with 1 fat one should not pass. Minor: - Budget trimmed to 3.5–4 hrs L40S (was 6–10) - L40S pool made explicit as target - Phase 1 branches now pushed as wip/* for crash safety - Hyperopt-params mismatch handling documented --- .../specs/2026-04-21-policy-quality-design.md | 47 ++++++++++++++----- 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-design.md b/docs/superpowers/specs/2026-04-21-policy-quality-design.md index de110ec63..93beb8ac1 100644 --- a/docs/superpowers/specs/2026-04-21-policy-quality-design.md +++ b/docs/superpowers/specs/2026-04-21-policy-quality-design.md @@ -82,7 +82,15 @@ main timeline: - **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production. - **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence. -**Budget estimate:** ~3–5 days elapsed. ~6–10 hours L40S GPU time total (≈1 hour per validation run, 2–3 iterations expected). +**Budget estimate:** ~3–5 days elapsed. + +- L40S validation runs: ~1 hour each, budget 3 runs = **3 hrs** (one after Phase 2, up to 2 more to iterate if partial) +- Local smoke runs (RTX 3050): ~3 min each, budget unlimited +- Investigation HEALTH_DIAG runs during Phase 1 use the same Argo workflows, 20-epoch × 2-fold = ~12 min on L40S, budget 3 = ~30 min + +Total GPU: **~3.5–4 hrs L40S** (not 6–10 — earlier estimate was conservative). + +**Target GPU pool:** `ci-training-l40s` throughout. H100 only if L40S queue saturates (no expected architecture-specific differences). --- @@ -100,9 +108,13 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga - HEALTH_DIAG readback: per-magnitude Q means, var_scale mean, kelly_f mean, per-branch grad ratio, action distribution per bin. **`reward_component_audit.rs`** — Track 2 gate -- Train 20 epochs, record per-term `|contribution| / |total step_return|` for each of the 8 reward terms (R1–R8 enumerated below). -- Assert: each ACTIVE term has `|contribution|/|total| ≥ 1%` in `≥ 50%` of epochs. -- Terms below threshold are triaged for DELETE in Phase 2. +- Train 20 epochs. +- Reward terms fall into three measurement classes, each with its own detection: + - **Additive** (R1 step_return, R5 micro-reward, R6 loss-aversion, R7 segment-patience): record `|term| / |total reward|` per step, assert the active term's median ratio ≥ 1% across epochs. + - **Transform** (R2 PopArt): record `|post_popart − pre_popart|`; if < 10% of |pre_popart| typical scale, PopArt is no-op-equivalent and DELETE-triage. + - **Sample-selector** (R3 CF-flip, R4 trailing-stop): record activation rate (fraction of samples/episodes the term fires on) AND downstream effect (grad-norm delta on fired vs unfired batches). Sub-noise = <1% firing rate OR indistinguishable grad signal. +- Terms below their class's threshold are triaged for DELETE in Phase 2. +- Implementation detail: the test uses one training run with instrumentation, not 8 separate runs. **`multi_fold_convergence.rs`** — fast local variant of the full gate - 3 folds × 20 epochs (vs 6 × 50 on L40S). Runs ~5 min on RTX 3050. @@ -132,7 +144,13 @@ Track 2 fields (reward V7 audit): ## 5. Phase 1 — Parallel investigation tracks -No commits land during Phase 1. Findings live in uncommitted worktree or throwaway local branches. +**Branch discipline during Phase 1:** + +- No commits land on `main`. +- Each track uses a throwaway local branch (`wip/policy-quality-track1`, `wip/policy-quality-track2`) that IS pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable. +- Track branches are never merged. At Phase 2, findings are re-applied by hand on a fresh `main` branch, and the throwaway branches are deleted. + +This gives us crash safety (branches pushed) + baseline cleanliness (nothing lands on main). ### 5.1 Track 1 — Magnitude collapse diagnosis @@ -140,10 +158,10 @@ Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** | # | Hypothesis | Detection signal | Detection threshold | |---|---|---|---| -| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | Per-bin reward mean + variance from replay buffer | Full Sharpe/state < 0.5× Quarter Sharpe/state across ≥10/20 epochs | +| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | During a forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | Ratio `mean(R_full) / std(R_full)` is ≤ 0.5× `mean(R_quarter) / std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. | | H2 | Var[Q] conviction-sizer over-shrinking — `effective_max_pos *= 1/(1 + sqrt(Var[Q]))` permanently collapses the target toward Quarter. | `var_scale_mean` field | `var_scale_mean < 0.3` across ≥10/20 epochs | | H3 | Kelly cap early-firing — Kelly stats (computed from Quarter-only history) cap at Quarter after 20+ trades. | `kelly_f_mean` field | `kelly_f_mean ≤ 0.25` across ≥10/20 epochs | -| H4 | Per-branch gradient under-allocation — magnitude head gets too small a share of the gradient budget. | `grad_ratio_mag_dir` field | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs | +| H4 | Magnitude-head gradient starvation — the per-action-branch gradient (*not* the per-loss-component IQN/CQL/C51/Ens split) to the magnitude head is much smaller than to the direction head. Magnitude weights stay near init. | `grad_ratio_mag_dir` field, measured as end-of-epoch average of per-step ‖∇ W[mag_head]‖ ÷ ‖∇ W[direction_head]‖ (pre-clip). | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs | | H5 | Bootstrap bias — initial random exploration anchored Quarter, epsilon decayed before the policy escaped. | Action entropy vs epsilon schedule | entropy < 0.3 before epoch 5 AND epsilon below 0.1 | **Fix sketch (if confirmed):** @@ -151,7 +169,7 @@ Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** - H1 → rebalance reward gradient per magnitude (advantage weighting by bin, or per-bin loss scale) - H2 → clamp `var_scale ≥ 0.5` OR remove entirely if redundant with Kelly - H3 → decouple Kelly from magnitude conditioning (use Kelly as position scaler, not magnitude quantizer) -- H4 → reallocate per-branch gradient budget (currently IQN=60%, CQL=25%, C51=10%, Ens=5% per ADR; may need explicit magnitude-head share) +- H4 → investigate why magnitude-head gradient is small. Candidates: magnitude-branch loss weight too low in the 4-branch advantage sum, magnitude head's `w_b1fc` dimensionality causing numerical suppression, VSN (variable selection network) masking magnitude features. Fix by rebalancing the 4-branch advantage weights OR adjusting initial weight scale for the magnitude head. (NOT the IQN/CQL/C51/Ens loss-component budget — that's orthogonal.) - H5 → extend exploration warmup for magnitude branch only ### 5.2 Track 2 — Reward V7 audit @@ -244,11 +262,13 @@ Existing walk-forward config produces 6 folds on `test_data/futures-baseline`. | Gate | Pass condition | Source | |---|---|---| -| A — Action-dist | `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds | HEALTH_DIAG `action_dist_*`, absence of LOW EXPOSURE DIVERSITY warnings | -| A — WinRate | Val WinRate in [55%, 65%] across folds | Metrics kernel | -| A — Trade count | ≥100 trades per val window (avg across 6 folds) | Metrics kernel | -| B — Multi-fold Sharpe | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log | -| B — Multi-fold WinRate | Val WinRate > 55% on ≥5/6 folds | Metrics kernel | +| A — Action-dist (soft) | `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds | HEALTH_DIAG `action_dist_*`, absence of LOW EXPOSURE DIVERSITY warnings | +| A — WinRate (soft) | Val WinRate ≥ 55% across folds (upper-bound fine) | Metrics kernel | +| A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel | +| **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log | +| **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel | + +Criterion B gates MUST pass for the spec to close successfully. Criterion A gates are soft — if B passes but some A gate misses, the spec closes with a documented A follow-up. ### 7.3 Partial-success handling @@ -278,6 +298,7 @@ These are the regression coverage for the invariants this spec establishes. - Hyperopt re-runs → separate concern; current defaults are used as baseline - Architectural changes beyond action space (no new network heads, no new layers) - State representation changes (OFI features, regime features — kept as-is) +- Hyperopt-params file regeneration. If Phase 2 changes action-space layout (e.g., adds a 4th magnitude bin), the existing `hyperopt_params.json` may have stale branch sizes. Policy: detect mismatch at runtime, log warning, use CLI branch-size defaults. A proper hyperopt re-run is a separate concern (sub-project B includes it implicitly as part of multi-fold iteration). ---