diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-design.md b/docs/superpowers/specs/2026-04-21-policy-quality-design.md index 26964667c..3f22d5954 100644 --- a/docs/superpowers/specs/2026-04-21-policy-quality-design.md +++ b/docs/superpowers/specs/2026-04-21-policy-quality-design.md @@ -34,20 +34,26 @@ Paper trading integration, canary deployment, hyperopt re-runs, and architecture ## 2. Success criteria -The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes both criteria: +The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the `feat/policy-quality` branch is merged to main via PR. -### 2.1 Criterion A — Action distribution +### 2.1 Mandatory gates (all must pass to merge) -- `F_Half ≥ 15%` AND `F_Full ≥ 15%` on at least **4/6 folds** -- Val `WinRate ∈ [55%, 65%]` across folds (above 65% is fine, but the bar is 55% floor) -- `≥ 100` trades per val window, averaged across the 6 folds +- **B-1 Multi-fold Sharpe:** Best val Sharpe (annualized) `> 10` on at least **5/6 folds** +- **B-2 Multi-fold WinRate:** Val WinRate `> 55%` on at least **5/6 folds** +- **SN Surrogate-noise:** Random-action Sharpe `≤ 0.5×` trained-model Sharpe on same val data (`surrogate_noise_check.rs`) -### 2.2 Criterion B — Multi-fold consistency +### 2.2 Soft gates (ship with documented follow-up if missed) -- Best val Sharpe (annualized) `> 10` on at least **5/6 folds** -- Val `WinRate > 55%` on at least **5/6 folds** +- **A-1 Action-dist:** `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds (*substitute for H9-confirmed case in §7.2*) +- **A-2 WinRate range:** Val WinRate `≥ 55%` across folds +- **A-3 Trade count:** `≥ 100` trades per val window on **≥5/6 folds** (per-fold, not averaged) +- **CA Controller-activity:** No single controller fires in >50% of epochs — soft; load-bearing controllers go to sub-project B tech debt -Failing a gate does not trigger a revert. A follow-up spec (`docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md`) documents what blocked it. +### 2.3 Outcome paths + +- **All mandatory + all soft pass** → merge to main, close spec clean. +- **All mandatory pass, some soft miss** → merge to main, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. +- **Any mandatory miss** → `feat/policy-quality` stays unmerged. Up to 2 more iterations on the feature branch. If still failing, scope exceeds this spec — escalate to sub-project A v2. --- @@ -107,7 +113,9 @@ Total GPU: **~5.5–6.5 hrs L40S**. ## 4. Phase 0 — Measurement substrate -Single commit lands on main. Adds smoke tests + HEALTH_DIAG extensions. No behavioral change. +Single commit on `feat/policy-quality` (branch created from current `main` HEAD). Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds). + +Tag this commit `policy-quality-baseline` for rollback anchor. ### 4.1 New smoke tests (ignored, run manually) @@ -171,9 +179,9 @@ Track 4 fields (exploration V7 audit): ### 4.3 Phase 0 exit criteria -- All new smoke tests compile and pass against the pre-fix baseline (they will fail their PASS assertions — the goal is that they RUN, so we have measurements). -- HEALTH_DIAG fields appear in training logs with non-zero values. -- Baseline reference metrics recorded in a short follow-up note (not committed — investigation-phase material). +- All new smoke tests compile and produce output. The Phase 0 smokes use **relaxed thresholds** (e.g. magnitude ≥5% not 15%) so they pass current code; they will tighten in later commits or their production equivalent lives in the Phase 3 validation gate. +- HEALTH_DIAG fields appear in training logs with non-zero values across a baseline 20-epoch run. +- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to `feat/policy-quality` — audit trail, not investigation-only. --- @@ -329,11 +337,19 @@ fn test_exploration_covers_magnitude_space() { Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item. Four track outputs consolidate into one Phase 2 commit. +**Conflict resolution when tracks disagree:** + +Tracks can reach conflicting conclusions — e.g. Track 1-H6 might say "fix trailing stop for magnitude" while Track 2 measures trailing stop contribution as sub-noise and triages DELETE. Conflict-resolution order: + +1. **Mandatory gate dominance.** A fix that removes an item blocks a mandatory-gate-relevant mechanism only if measurement shows the mechanism isn't contributing to the mandatory outcome. In the trailing-stop example: if Track 2 shows R4 is sub-noise, DELETE wins regardless of Track 1 wanting to fix it — "fix a dead thing" is pointless. +2. **Simplification wins.** Among equally-justified options, pick the one that reduces code surface (DELETE > FIX > ADD). +3. **Escalation.** If (1) and (2) don't resolve, the spec author decides and documents the call in the Phase 2 commit. + --- ## 6. Phase 2 — Convergence commit -Single commit on main. Lands all Phase 1 findings together. +Single commit on `feat/policy-quality`. Lands all Phase 1 findings together. Phase-4 merge to main happens only after Phase-3 validation passes — see §7. **Commit structure:** @@ -371,8 +387,8 @@ Validation gate (Phase 3): **Rules:** -- **Single commit.** Any bug discovered during Phase 2 integration testing goes into the same commit via amend. -- **All smoke tests green** (td_propagation + new Track 1/2 smokes + `test_50_epoch_convergence` from `training_stability.rs`) before push. +- **Single commit on `feat/policy-quality`.** Any bug discovered during Phase 2 integration testing goes into the same commit via amend. +- **All smoke tests green before push:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`). - **No config-file-only changes.** Every behavioral change has source code attached. --- @@ -403,15 +419,29 @@ Criterion B gates AND surrogate-noise check MUST pass for the spec to close succ **Special case for H9 confirmed (magnitude branch deleted):** if Phase 1 H9 confirms and Phase 2 removes the magnitude branch, Criterion A's `F_Half ≥ 15%` and `F_Full ≥ 15%` become vacuous (no bins to distribute across). Replace with substitute gate: trade count AND WinRate criteria still apply; the action-distribution gate becomes `direction bin distribution healthy` (F_Short + F_Long each ≥ 20% of non-Flat actions). The spirit of Criterion A — "model actually trades, doesn't collapse to a single action" — is preserved. -### 7.3 Partial-success handling +### 7.3 Outcome handling -If any gate misses, the Phase 2 commit is NOT reverted. A follow-up note lands at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: +See §2.3 for the outcome paths. Concretely: -- Which gate missed and by how much -- What hypothesis (if any) was wrongly rejected in Phase 1 -- Proposed next-spec scope +- **Missed mandatory gate** → `feat/policy-quality` stays unmerged. Iterate up to 2 more validation runs with additional commits on the feature branch. If 3 runs total still miss a mandatory gate, escalate out of this spec. +- **Missed soft gate only** → merge to main via `--no-ff` PR; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope. -### 7.4 Permanent artifacts (survive spec closure) +### 7.4 Phase 4 — Merge to main + +```bash +# After mandatory gates pass: +git checkout main +git merge --no-ff feat/policy-quality # preserves branch identity in history +git push origin main +git tag policy-quality-v1 # named checkpoint +git branch -d feat/policy-quality # local cleanup +git push origin --delete feat/policy-quality +# Leave wip/pq-track{1..4} branches for 30 days as investigation audit trail, then delete. +``` + +`--no-ff` is deliberate — the merge-commit documents "this is where the 4-track V7 audit landed" in the history, even if the branch is linear. + +### 7.5 Permanent artifacts (survive spec closure) Left in the tree after the spec closes: @@ -448,7 +478,7 @@ These are the regression coverage for the invariants this spec establishes. | Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. | | Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Feature branch stays unmerged. `feat/policy-quality` kept for rerun attempts. No pollution of main. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). | | Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. | -| Retraining cost runs beyond 10 hrs L40S | Low | Low | Iteration budget capped at 3 full validation runs. If still failing, triage with faster subset runs. | +| Iteration cost exceeds ~7 hrs L40S (budgeted 5.5–6.5 for plan + 1 buffer) | Low | Low | Iteration budget hard-capped at 3 Phase-3 validation runs (3 hrs). If the 3rd still misses a mandatory gate, the spec escalates — no further validation runs on this branch. | | Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. | | Controller-audit (Track 3) finds multiple load-bearing controllers that can't be removed in this spec | Medium | Medium | Document as tech-debt, flag for sub-project B. Spec can still close if policy is profitable. But "load-bearing controllers = not production-deployable" remains a known concern. |