From 944fecd913a2966c6f1a6997d182d316d8185fc9 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Tue, 21 Apr 2026 21:06:33 +0200 Subject: [PATCH] =?UTF-8?q?docs(policy-quality):=20main-branch=20workflow?= =?UTF-8?q?=20=E2=80=94=20cut=20feat/wip=20branches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User preference: everything lands on main, ~500 LOC is small enough that feature-branch isolation isn't needed. Spec changes: * §2.3 outcome paths: drop "unmerged branch" vocabulary * §3 architecture: main-branch timeline, no feat/, no wip/ * §4 Phase 0: commits on main, tag policy-quality-baseline at end * §5 Phase 1: experiments are worktree edits reverted after, only triage docs commit (to main) * §6 Phase 2: commits land on main, author chooses granularity, each leaves smokes green * §7.3 outcome handling: iterate on main, baseline tag for rollback * §7.4 closing: tag policy-quality-v1, update status Plan changes: * Task 0.1: verify-starting-state (not branch creation) * Task 1.x: temp-edit-then-revert (not wip branches) * Task 1.5: triage docs commit directly to main * Phase 4: tag + close (not merge + cleanup) * Rollback: one git reset --hard command --- .../plans/2026-04-21-policy-quality.md | 292 +++++++----------- .../specs/2026-04-21-policy-quality-design.md | 91 +++--- 2 files changed, 150 insertions(+), 233 deletions(-) diff --git a/docs/superpowers/plans/2026-04-21-policy-quality.md b/docs/superpowers/plans/2026-04-21-policy-quality.md index 481199626..94b139910 100644 --- a/docs/superpowers/plans/2026-04-21-policy-quality.md +++ b/docs/superpowers/plans/2026-04-21-policy-quality.md @@ -4,7 +4,7 @@ **Goal:** Execute the 4-phase V7 audit specified in `docs/superpowers/specs/2026-04-21-policy-quality-design.md` — build measurement substrate, run parallel investigation tracks, land all confirmed fixes in a single convergence commit, validate on 6-fold × 50-epoch L40S, merge to main. -**Architecture:** Feature branch `feat/policy-quality` isolates work from main. Phase 0 commit + Phase 2 commit + Phase 3 fixup commits on the feature branch. Merge to main via `--no-ff` PR only after MANDATORY gates (B-Sharpe, B-WinRate, surrogate-noise) pass. Failures don't pollute main. +**Architecture:** All commits land on `main`. Tagged `policy-quality-baseline` after Phase 0 as rollback anchor. Phase 2 fixes commit incrementally — each leaves main working. Closing tag `policy-quality-v1` on the final validated commit. **Tech Stack:** Rust (workspace crates: ml, ml-dqn, ml-core), CUDA (cuBLAS, custom kernels via nvcc), Argo Workflows on Scaleway Kapsule, L40S GPU (sm_89), fxcache binary feature cache, `#[ignore]` smoke tests gated on `FOXHUNT_TEST_DATA`. @@ -73,7 +73,7 @@ Per-finding. Recorded during Phase 2 planning commit. **Exit criteria:** All 6 new smoke tests compile, run, and produce readable output. HEALTH_DIAG fields appear in training logs with non-zero values. **Phase 0 commits (in order):** -1. Create feature branch + tag baseline +1. Verify starting state 2. HEALTH_DIAG field scaffolding (host-side, zero-returning stubs) 3. Track 1 HEALTH_DIAG wiring (GPU-side readbacks for H1–H10 metrics) 4. Track 2 HEALTH_DIAG wiring (reward-term contribution) @@ -84,41 +84,25 @@ Per-finding. Recorded during Phase 2 planning commit. --- -### Task 0.1: Create feature branch and baseline tag +### Task 0.1: Verify starting state **Files:** -- Modify: git only +- git only -- [ ] **Step 1: Verify main is at expected HEAD** - -Run: `git fetch origin && git log --oneline -1 origin/main` -Expected: output shows `ad677466c design(policy-quality): revision 4 — functional gaps` or later. - -- [ ] **Step 2: Create feature branch from main** +- [ ] **Step 1: Verify on main, up to date** Run: ```bash git checkout main -git pull --ff-only -git checkout -b feat/policy-quality +git pull --ff-only origin main +git log -1 --oneline ``` -Expected: `Switched to a new branch 'feat/policy-quality'` +Expected: clean working tree, at latest main (spec/plan commits present). -- [ ] **Step 3: Push branch to origin with upstream tracking** +- [ ] **Step 2: Note the baseline SHA** -Run: `git push -u origin feat/policy-quality` -Expected: `branch 'feat/policy-quality' set up to track 'origin/feat/policy-quality'.` - -- [ ] **Step 4: Create baseline tag (will move to Phase-0-final commit at end of Phase 0)** - -Run: -```bash -# Tag the CURRENT HEAD — we'll move this tag after Phase 0 completes. -# For now it's an anchor reference. -git tag -f policy-quality-baseline HEAD -git push -f origin policy-quality-baseline -``` -Expected: tag pushed, `git tag -l policy-quality-baseline` lists it. +Run: `git rev-parse HEAD | tee /tmp/policy-quality-pre-phase0-sha` +Expected: one SHA. Kept for reference — we'll tag the Phase-0-final commit (not this SHA) as `policy-quality-baseline` at Task 0.17. --- @@ -1265,53 +1249,30 @@ git commit -m "docs(policy-quality): Phase 0 baseline metrics (pre-audit referen git push ``` -- [ ] **Step 4: Move the `policy-quality-baseline` tag to the Phase 0 head** +- [ ] **Step 4: Tag the final Phase-0 commit as `policy-quality-baseline`** ```bash -git tag -f policy-quality-baseline HEAD -git push -f origin policy-quality-baseline +git tag -a policy-quality-baseline -m "Phase 0 complete — instrumentation in, pre-fix reference" +git push origin policy-quality-baseline ``` +This tag is the rollback anchor for Phases 2+. If something breaks on main, `git reset --hard policy-quality-baseline` restores this known-good state. + --- -## Phase 1 — Parallel Investigation Tracks +## Phase 1 — Investigation Tracks **Goal:** run each track's HEALTH_DIAG-instrumented training, collect evidence, classify each hypothesis/term/controller/mechanism as CONFIRMED or REJECTED. -**Exit criteria:** Triage document per track committed to the corresponding `wip/pq-track*` branch. Track outputs merged into a single Phase 2 planning document. +**Exit criteria:** 4 triage documents committed to main. Phase-2 implementation plan written from consolidated findings. -**Branch discipline:** each track lives on `wip/pq-track{1,2,3,4}-*`. Nothing lands on `feat/policy-quality` during Phase 1. - ---- - -### Task 1.0: Create investigation branches - -- [ ] **Step 1: From `feat/policy-quality`, create 4 wip branches** - -```bash -git checkout feat/policy-quality -for track in 1-magnitude 2-reward 3-controllers 4-exploration; do - git checkout -b wip/pq-track${track} - git push -u origin wip/pq-track${track} -done -git checkout feat/policy-quality -``` - -Each wip branch is a parallel universe for investigation. Changes here don't land on `feat/policy-quality`. +**Discipline:** some hypotheses need temporary code edits (e.g., force epsilon=1.0 for H1). Edit the file, run the experiment, then `git checkout -- ` to revert. Don't commit experimental code. Only triage docs commit. --- ### Task 1.1: Track 1 — Magnitude investigation -**Branch:** `wip/pq-track1-magnitude` - -- [ ] **Step 1: Checkout track branch** - -```bash -git checkout wip/pq-track1-magnitude -``` - -- [ ] **Step 2: Baseline run — inherits Phase 0 instrumentation** +- [ ] **Step 1: Baseline run — inherits Phase 0 instrumentation** Already have HEALTH_DIAG from Phase 0. Run 20-epoch × 2-fold on L40S: @@ -1324,18 +1285,25 @@ Wait for completion, dump logs: argo logs -n foxhunt > /tmp/track1-baseline.log ``` -- [ ] **Step 3: Forced-exploration experiment (for H1 detection)** +- [ ] **Step 2: Forced-exploration experiment (for H1 detection)** -H1 requires epsilon=1.0 for 1 epoch at mid-training. Edit `training_loop.rs` temporarily to force epsilon=1.0 at epoch 10, re-run workflow, dump logs as `/tmp/track1-forced-explore.log`. Do NOT commit the temporary change. +H1 requires epsilon=1.0 for 1 epoch at mid-training. -- [ ] **Step 4: Classify each hypothesis** +1. Edit `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — force `epsilon = 1.0` when `epoch == 10`. +2. Run the workflow: + ```bash + ./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline + argo logs -n foxhunt > /tmp/track1-forced-explore.log + ``` +3. Revert the edit: `git checkout -- crates/ml/src/trainers/dqn/trainer/training_loop.rs` + +- [ ] **Step 3: Classify each hypothesis** For each of H1–H10, check the detection signal against the threshold per spec §5.1. Record CONFIRMED or REJECTED with evidence in `docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md`: ```markdown # Track 1 — Magnitude Collapse Triage -**Branch:** wip/pq-track1-magnitude **Log refs:** /tmp/track1-baseline.log, /tmp/track1-forced-explore.log ## H1 — Reward gradient asymmetry @@ -1348,7 +1316,8 @@ For each of H1–H10, check the detection signal against the threshold per spec (...repeat for H2–H10) ``` -Commit to the wip branch: +- [ ] **Step 4: Commit triage doc to main** + ```bash git add docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md git commit -m "triage(track1): magnitude hypotheses H1-H10 verdicts with evidence" @@ -1359,105 +1328,108 @@ git push ### Task 1.2: Track 2 — Reward V7 audit -**Branch:** `wip/pq-track2-reward` - -- [ ] **Step 1: Checkout + baseline run** +- [ ] **Step 1: Baseline run** ```bash -git checkout wip/pq-track2-reward ./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline +argo logs -n foxhunt > /tmp/track2-baseline.log ``` +Phase 0 reward-contrib HEALTH_DIAG fields already capture per-term signals. + - [ ] **Step 2: For each of R1–R8, apply the V7 audit template** Per spec §5.2: 1. Identify signal (git log, grep for the term, read its introducing commit) 2. Check better-form (does another term cover the same invariant?) -3. Measure (reward_contrib_* field from HEALTH_DIAG over 20 epochs) +3. Measure (`reward_contrib_*` field from `/tmp/track2-baseline.log`) 4. Decide per the decision matrix -- [ ] **Step 3: Record triage** +- [ ] **Step 3: Record triage + commit** -Write `docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md`: +Write `docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md` (one section per R1–R8) and commit to main: -```markdown -# Track 2 — Reward V7 Audit Triage - -(one section per R1–R8 with identify / better-form / measured / decision) +```bash +git add docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md +git commit -m "triage(track2): reward V7 audit R1-R8 verdicts" +git push ``` -Commit to wip branch. - --- ### Task 1.3: Track 3 — Controllers V7 audit -**Branch:** `wip/pq-track3-controllers` +- [ ] **Step 1: Use the Phase 0 HEALTH_DIAG controller-fire-rate data (already captured)** -- [ ] **Step 1: Checkout + baseline run** +Controller fire rates appear in every HEALTH_DIAG line. Extract from `/tmp/track2-baseline.log` or run another 20-epoch if needed. -Same as 1.2 but focused on controller fire rates. +- [ ] **Step 2: Disable-and-measure experiments** -- [ ] **Step 2: Disable-and-measure experiment** +For candidates flagged "CANDIDATE FOR DELETE" per spec §5.3 (10–50% fire rate), disable each temporarily and compare outcomes: -For candidates flagged as "CANDIDATE FOR DELETE" (10–50% fire rate), temporarily disable each one (gate with a flag), retrain, compare outcomes. Record delta. +1. Edit `training_loop.rs` — change the controller's if-block to `if false`. +2. `./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline` +3. Record Best Sharpe + grad stability vs baseline in the triage doc. +4. Revert: `git checkout -- crates/ml/src/trainers/dqn/trainer/training_loop.rs` +5. Repeat for next candidate. -- [ ] **Step 3: Triage document** +- [ ] **Step 3: Triage document + commit** -Write `docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md`, one section per C1–C7. Commit. +Write `docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md`, one section per C1–C7. Commit to main. --- ### Task 1.4: Track 4 — Exploration V7 audit -**Branch:** `wip/pq-track4-exploration` +- [ ] **Step 1: Use Phase 0 HEALTH_DIAG data** -- [ ] **Step 1: Baseline run (from Phase 0)** - -HEALTH_DIAG already captures `explore_entropy_*` and `noisy_net_sigma_mean`. No additional instrumentation needed. +`explore_entropy_*` and `noisy_sigma_mean` are already captured. No additional instrumentation needed for E1/E2/E4. - [ ] **Step 2: For each E1–E5, apply V7 template** -Count-based bonus (E3) and entropy regularization (E4) may be redundant with NoisyNets (E2). Measure each mechanism's contribution to per-epoch action-entropy. Run with each mechanism disabled in turn (5 short runs). +Count-based bonus (E3) and entropy regularization (E4) may be redundant with NoisyNets (E2). For each mechanism: +1. Edit its enable-block to `if false` in training_loop.rs +2. Run 20-epoch +3. Record impact on magnitude entropy +4. `git checkout -- `, move to next -- [ ] **Step 3: Triage document** +- [ ] **Step 3: Triage document + commit** -Write `docs/superpowers/specs/2026-04-21-policy-quality-track4-triage.md`. Commit. +Write `docs/superpowers/specs/2026-04-21-policy-quality-track4-triage.md`. Commit to main. --- -### Task 1.5: Consolidate 4 triage documents into Phase 2 planning +### Task 1.5: Write Phase 2 implementation plan -- [ ] **Step 1: Back on `feat/policy-quality`** +- [ ] **Step 1: Consolidate findings** + +All 4 triage docs are on main. Skim each, compile a list of CONFIRMED findings. + +- [ ] **Step 2: Invoke writing-plans against the consolidated triage** ```bash -git checkout feat/policy-quality +# Use superpowers:writing-plans with input: +# "Given the 4 triage docs at +# docs/superpowers/specs/2026-04-21-policy-quality-track{1,2,3,4}-triage.md +# produce a Phase 2 implementation plan with concrete TDD tasks per CONFIRMED +# finding. Target: docs/superpowers/plans/2026-04-21-policy-quality-phase2.md" ``` -- [ ] **Step 2: Cherry-pick the four triage documents** +- [ ] **Step 3: Commit the Phase 2 plan** ```bash -for track in 1 2 3 4; do - git checkout wip/pq-track${track}-* -- docs/superpowers/specs/2026-04-21-policy-quality-track${track}-triage.md -done -git add docs/superpowers/specs/2026-04-21-policy-quality-track*-triage.md -git commit -m "triage(policy-quality): consolidate 4-track triage docs to feat branch" +git add docs/superpowers/plans/2026-04-21-policy-quality-phase2.md +git commit -m "plan(policy-quality): Phase 2 implementation plan from triage findings" git push ``` -- [ ] **Step 3: Write Phase 2 implementation plan** - -Based on the triage docs, write `docs/superpowers/plans/2026-04-21-policy-quality-phase2.md` with concrete TDD tasks for each CONFIRMED finding. This is a separate planning pass — invoke the `writing-plans` skill against the consolidated triage. - -Commit the Phase 2 plan. - --- ## Phase 2 — Convergence Fix -**Goal:** implement all fixes identified in Phase 1 triage, in a single commit on `feat/policy-quality`. Every new smoke test (Track 1–4) passes at its production threshold, not just the relaxed smoke-gate threshold. +**Goal:** implement all fixes identified in Phase 1 triage on main. Each commit leaves main working + smokes green. -**Exit criteria:** all 6 Phase 0 smokes pass; existing smokes (td_propagation, test_50_epoch_convergence) still pass. +**Exit criteria:** all 6 Phase 0 smokes pass at their PRODUCTION thresholds (tighten from Phase 0's relaxed smoke thresholds). Existing smokes (`td_propagation`, `test_50_epoch_convergence`) still pass. **Template for Phase 2 tasks (filled in by the Phase 2 plan written in Task 1.5):** @@ -1465,14 +1437,10 @@ For each CONFIRMED finding in any of the 4 triages: 1. Write test asserting the fixed behaviour (e.g., magnitude bin usage at production threshold) 2. Implement the fix per the spec's fix sketch 3. Run all 6 Phase 0 smokes locally — expect all pass -4. Amend into the single Phase 2 convergence commit +4. Commit with message referencing the triage finding (e.g., `fix(dqn): H2 — clamp var_scale ≥ 0.5 per track1 triage`) +5. Push -Single final commit message: -``` -fix(dqn): policy-quality V7 audit — bugs, dead terms removed - -(full enumeration per spec §6 commit structure) -``` +Commits can be one-per-finding (preferred — easier to review) or grouped by track. Author's choice. --- @@ -1482,7 +1450,7 @@ fix(dqn): policy-quality V7 audit — bugs, dead terms removed ### Task 3.1: Run validation workflow -- [ ] **Step 1: Ensure `feat/policy-quality` HEAD is pushed and committed by Phase 2** +- [ ] **Step 1: Ensure main HEAD has Phase 2 fixes committed and pushed** Run: `git status && git log -1 --oneline` Expected: clean, last commit is the Phase 2 convergence. @@ -1571,7 +1539,7 @@ Expected: pass — trained pooled Sharpe exceeds 95th %ile of N=30 surrogates. Commit `phase3-results.md` and proceed to merge. -- [ ] **Step 2b: If any mandatory fails → iterate on `feat/policy-quality`** +- [ ] **Step 2b: If any mandatory fails → iterate on main** Up to 2 additional iterations total (max 3 Phase-3 validation runs). Each iteration: - Diagnose from log which gate failed and why @@ -1583,118 +1551,68 @@ Up to 2 additional iterations total (max 3 Phase-3 validation runs). Each iterat --- -## Phase 4 — Merge +## Phase 4 — Close -**Goal:** merge `feat/policy-quality` to main, tag release, clean up branches. +**Goal:** tag `policy-quality-v1` on the final validated main commit, update the spec status, close. **Prerequisites:** Phase 3 all MANDATORY gates pass. -### Task 4.1: Final pre-merge verification +### Task 4.1: Tag the validated commit -- [ ] **Step 1: Fast-forward main on local** +- [ ] **Step 1: Confirm on main at the post-Phase-3 HEAD** ```bash git checkout main -git pull --ff-only origin main +git log -1 --oneline ``` -- [ ] **Step 2: Confirm feat branch is up to date** +- [ ] **Step 2: Tag** ```bash -git checkout feat/policy-quality -git log origin/main..HEAD --oneline -``` +git tag -a policy-quality-v1 -m "Policy quality V7 audit complete — sub-project A closed. -Expected: all Phase 0 through Phase 3 commits listed, nothing unexpected. - -### Task 4.2: Merge via `--no-ff` - -- [ ] **Step 1: Return to main, merge feat branch** - -```bash -git checkout main -git merge --no-ff feat/policy-quality -m "Merge feat/policy-quality: policy-quality V7 audit - -Sub-project A of production-readiness roadmap complete. All MANDATORY gates passed in Phase 3 validation: - B-Sharpe: Best val Sharpe > 10 on /6 folds - B-WinRate: Val WinRate > 55% on /6 folds - Surrogate-noise: trained pooled Sharpe beyond 95th %ile of 30 random surrogates -Phase 2 convergence commit: -Phase 3 validation results: docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md -" -``` - -- [ ] **Step 2: Push main** - -```bash -git push origin main -``` - -### Task 4.3: Tag + cleanup - -- [ ] **Step 1: Tag release** - -```bash -MERGE_COMMIT=$(git rev-parse HEAD) -git tag -a policy-quality-v1 -m "Policy quality V7 audit complete — sub-project A closed" "$MERGE_COMMIT" +Phase 3 results: docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md" git push origin policy-quality-v1 ``` -- [ ] **Step 2: Delete local feature branch** +### Task 4.2: Update spec status + roadmap -```bash -git branch -d feat/policy-quality -git push origin --delete feat/policy-quality -``` +- [ ] **Step 1: Add CLOSED header to the spec** -- [ ] **Step 3: Keep wip/pq-track* branches for 30 days as audit trail** - -Set a calendar reminder (or open a follow-up task) to run in 30 days: - -```bash -for track in 1-magnitude 2-reward 3-controllers 4-exploration; do - git push origin --delete wip/pq-track${track} -done -``` - -### Task 4.4: Close spec + update roadmap - -- [ ] **Step 1: Append closure note to the spec** - -Add at top of `docs/superpowers/specs/2026-04-21-policy-quality-design.md`: +At top of `docs/superpowers/specs/2026-04-21-policy-quality-design.md`, insert: ```markdown -**Status (2026-MM-DD): CLOSED — merged to main as `policy-quality-v1`.** -**Phase 3 results:** see `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md` +**Status (2026-MM-DD): CLOSED — tagged `policy-quality-v1`.** +**Phase 3 results:** `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md` ``` -- [ ] **Step 2: Update production-readiness roadmap** - -If sub-project roadmap doc exists, check off sub-project A. Note unblocking of sub-projects B, C. - -- [ ] **Step 3: Commit + push** +- [ ] **Step 2: Commit + push** ```bash git add docs/superpowers/specs/2026-04-21-policy-quality-design.md -git commit -m "docs(policy-quality): mark sub-project A closed, merged as policy-quality-v1" +git commit -m "docs(policy-quality): mark sub-project A closed, tagged policy-quality-v1" git push ``` +Sub-projects B, C, D (validation rigor, paper trading, production ops) are now unblocked. + --- ## Rollback Procedure (emergency) -If `policy-quality-v1` causes production issues: +If a commit on main breaks something that wasn't caught by smokes: ```bash -git checkout main git reset --hard policy-quality-baseline -git push -f origin main # only if team coordination allows +git push -f origin main # only with team coordination ``` -The baseline tag points at the Phase 0 commit — the last known-good state before any Phase 2 behavioural change. Recovery is one command. +The baseline tag = last Phase 0 commit = pre-fix reference state. Recovery is one command. --- diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-design.md b/docs/superpowers/specs/2026-04-21-policy-quality-design.md index d70ba2f23..430e033ca 100644 --- a/docs/superpowers/specs/2026-04-21-policy-quality-design.md +++ b/docs/superpowers/specs/2026-04-21-policy-quality-design.md @@ -34,7 +34,7 @@ Paper trading integration, canary deployment, hyperopt re-runs, and architecture ## 2. Success criteria -The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the `feat/policy-quality` branch is merged to main via PR. +The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the final main commit is tagged `policy-quality-v1`. ### 2.1 Mandatory gates (all must pass to merge) @@ -51,52 +51,53 @@ The spec closes when a single L40S validation run (6-fold × 50-epoch, the exist ### 2.3 Outcome paths -- **All mandatory + all soft pass** → merge to main, close spec clean. -- **All mandatory pass, some soft miss** → merge to main, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. -- **Any mandatory miss** → `feat/policy-quality` stays unmerged. Up to 2 more iterations on the feature branch (total 3 Phase-3 validation runs). If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with: +- **All mandatory + all soft pass** → tag `policy-quality-v1`, close spec clean. +- **All mandatory pass, some soft miss** → tag `policy-quality-v1`, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. Does not block sub-project B. +- **Any mandatory miss** → iterate on main (up to 2 more iterations, total 3 Phase-3 validation runs). Use `policy-quality-baseline` tag to revert if an iteration makes things worse. If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with: - Summary of the 3 failed runs' evidence (which gates, by how much, which hypotheses correlate with failure) - Explicit diagnosis: policy/reward/state bottleneck? architecture bottleneck? data bottleneck? - - Decision: continue on `feat/policy-quality` with new hypotheses, OR fork a new branch `feat/policy-quality-v2` with a different angle (e.g., reward redesign, architecture change, alternative algorithm PPO/SAC) - - Triage spec is a full brainstorming cycle (not a follow-up note) — restarts the skill flow. + - Decision: continue on main with new hypotheses, OR scope a different angle (reward redesign, architecture change, alternative algorithm PPO/SAC) — the triage spec is a full brainstorming cycle that restarts the skill flow. --- ## 3. Architecture -Feature-branch–isolated. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that merge into one convergence commit on the feature branch, validated, then merged to main via PR. +Main-branch only. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that converge into the Phase-2 fix commit set. Tagged baseline for rollback. ``` -feat/policy-quality branch: +main branch: - [Phase 0] commit: measurement substrate. Tagged policy-quality-baseline. - ↓ (baseline run against this commit — reference metrics frozen) + [Phase 0] commit(s): measurement substrate. Final commit tagged + ↓ policy-quality-baseline. Baseline metrics captured here. - [Phase 1] NO commits on feat/*. Four parallel investigation tracks live - on throwaway branches wip/pq-track{1,2,3,4}, pushed to origin - for crash safety. + [Phase 1] Investigation. Temporary experiments in local worktree / + git-stash — not committed (experiments are throwaway). Final + output is triage documents committed to main. ↓ - [Phase 2] commit: single convergence fix on feat/policy-quality. - All confirmed bugs across all 4 tracks fix together. + [Phase 2] commit(s): convergence fixes. Can be one commit or several + per confirmed finding — main is our working branch. Total + ~500 LOC expected; commit granularity is author's choice. ↓ - [Phase 3] commit(s) on feat/*: validation tweaks from 6-fold run. + [Phase 3] commit(s): validation iteration tweaks from 6-fold L40S run. Success-criteria gate checked. ↓ - [Phase 4] Merge feat/policy-quality → main via PR. Only on all gates green. - If validation fails, feature branch stays, main untouched. + [CLOSE] Tag policy-quality-v1 on the final validated commit. + If validation fails after 3 iterations, open triage spec + per §2.3. ``` **Key architectural decisions:** -- **Feature branch for isolation.** Every change lives on `feat/policy-quality` until Phase 4 merge. `main` stays clean until validation passes. Revert is "don't merge." +- **Main-branch workflow.** All commits land on main. ~500 LOC total is small enough that feature-branch isolation isn't needed. - **Measurement first.** No fix lands until its detection is codified in a smoke test. -- **No mid-investigation commits on feature branch.** Phase 1 is exploratory; only the convergence commit lands on `feat/policy-quality`. -- **All confirmed findings fix together.** No ranking, no "most likely cause" — V7 discipline gates inclusion (measurement confirms a real bug), every confirmed bug ships in Phase 2. +- **Phase 1 investigation stays local.** Experimental instrumentation lives in the worktree during measurement runs, never committed. Only the resulting triage documents commit to main. +- **Fixes can be multiple commits.** The spec's earlier "single convergence commit" rule is relaxed — each confirmed finding can be its own commit if that's easier to review. The discipline that remains: all fixes land before Phase 3 validation runs; each commit leaves smokes green. - **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production. - **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence. -- **Tagged baseline for rollback.** `git tag policy-quality-baseline` at Phase 0. If merged-to-main code ever regresses, `git reset --hard policy-quality-baseline` restores known state. +- **Tagged baseline for rollback.** `git tag policy-quality-baseline` after Phase 0 completes. If Phase 2+ breaks main, `git reset --hard policy-quality-baseline` restores known state. Tag `policy-quality-v1` marks the validated close. **Budget estimate:** ~4–6 days elapsed. @@ -117,9 +118,9 @@ Total GPU: **~5.5–6.5 hrs L40S**. ## 4. Phase 0 — Measurement substrate -Single commit on `feat/policy-quality` (branch created from current `main` HEAD). Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds). +One or more commits on `main`. Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds). -Tag this commit `policy-quality-baseline` for rollback anchor. +After Phase 0 completes, tag the final Phase-0 commit `policy-quality-baseline` for rollback anchor. ### 4.1 New smoke tests (ignored, run manually) @@ -197,7 +198,8 @@ Track 4 fields (exploration V7 audit): - All new smoke tests compile and produce output. The Phase 0 smokes use **relaxed thresholds** (e.g. magnitude ≥5% not 15%) so they pass current code; they will tighten in later commits or their production equivalent lives in the Phase 3 validation gate. - HEALTH_DIAG fields appear in training logs with non-zero values across a baseline 20-epoch run. -- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to `feat/policy-quality` — audit trail, not investigation-only. +- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to main — audit trail. +- `git tag policy-quality-baseline` on the final Phase-0 commit + `git push origin policy-quality-baseline`. --- @@ -205,11 +207,11 @@ Track 4 fields (exploration V7 audit): **Branch discipline during Phase 1:** -- No commits land on `main` OR on `feat/policy-quality`. -- Each track uses a throwaway local branch (`wip/pq-track1-magnitude`, `wip/pq-track2-reward`, `wip/pq-track3-controllers`, `wip/pq-track4-exploration`) pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable. -- Track branches are never merged. At Phase 2, findings are re-applied on `feat/policy-quality` as a single commit, and the wip branches are deleted. +- No **code** commits land on `main` during Phase 1 — the temporary instrumentation (e.g., forced-exploration epsilon=1.0 experiment, controller-disable experiments) is experimental and would pollute history. +- **Triage documents DO commit to `main`** as they're completed. Each track's triage doc is a permanent artifact. +- Experimental code changes live in the worktree (uncommitted) or in local `git stash` during a measurement run; discard after the triage doc is written. -This gives us crash safety (branches pushed) + feature-branch cleanliness (nothing lands on `feat/policy-quality` until Phase 2). +This is the pragma: instrumentation commits from Phase 0 are kept (useful), investigation code is throwaway, triage output is committed. ### 5.1 Track 1 — Magnitude collapse diagnosis @@ -363,9 +365,9 @@ Tracks can reach conflicting conclusions — e.g. Track 1-H6 might say "fix trai --- -## 6. Phase 2 — Convergence commit +## 6. Phase 2 — Convergence fix commits -Single commit on `feat/policy-quality`. Lands all Phase 1 findings together. Phase-4 merge to main happens only after Phase-3 validation passes — see §7. +Commits on `main` — one or several, author's choice. All Phase 1 findings fixed before Phase 3 validation begins. Each commit leaves smokes green. **Commit structure:** @@ -403,9 +405,10 @@ Validation gate (Phase 3): **Rules:** -- **Single commit on `feat/policy-quality`.** Any bug discovered during Phase 2 integration testing goes into the same commit via amend. -- **All smoke tests green before push:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`). +- **Commits land on main.** Author chooses granularity (one per finding, or grouped). Each commit leaves main in a working state. +- **All smoke tests green before push on every commit:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`). - **No config-file-only changes.** Every behavioral change has source code attached. +- **Commit messages reference the triage finding** (e.g., "fix(dqn): H2 — clamp var_scale ≥ 0.5 per track1 triage"). Makes git log readable as the audit trail. --- @@ -439,23 +442,19 @@ Criterion B gates AND surrogate-noise check MUST pass for the spec to close succ See §2.3 for the outcome paths. Concretely: -- **Missed mandatory gate** → `feat/policy-quality` stays unmerged. Iterate up to 2 more validation runs with additional commits on the feature branch. If 3 runs total still miss a mandatory gate, escalate out of this spec. -- **Missed soft gate only** → merge to main via `--no-ff` PR; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope. +- **Missed mandatory gate** → iterate up to 2 more validation runs with additional commits on main. If 3 runs total still miss a mandatory gate, escalate out of this spec (triage doc per §2.3). +- **Missed soft gate only** → tag `policy-quality-v1`; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope. -### 7.4 Phase 4 — Merge to main +### 7.4 Closing the spec + +When mandatory gates pass: ```bash -# After mandatory gates pass: -git checkout main -git merge --no-ff feat/policy-quality # preserves branch identity in history -git push origin main -git tag policy-quality-v1 # named checkpoint -git branch -d feat/policy-quality # local cleanup -git push origin --delete feat/policy-quality -# Leave wip/pq-track{1..4} branches for 30 days as investigation audit trail, then delete. +git tag -a policy-quality-v1 -m "Policy quality V7 audit — sub-project A closed" +git push origin policy-quality-v1 ``` -`--no-ff` is deliberate — the merge-commit documents "this is where the 4-track V7 audit landed" in the history, even if the branch is linear. +Update the top of this design doc with `Status: CLOSED`, reference the Phase-3 results doc, and mark sub-project A complete in any production-readiness tracking. ### 7.5 Permanent artifacts (survive spec closure) @@ -492,7 +491,7 @@ These are the regression coverage for the invariants this spec establishes. | Phase 1 finds all 10 Track-1 hypotheses REJECTED | Low | High | Track 2/3/4 almost certainly surface findings; if all 4 tracks come up empty, magnitude collapse might be **H9 (not a bug, data favors Quarter)**. Delete magnitude branch per H9 fix sketch — action-space simplification is still a production win. | | H9 confirmed: magnitude branch deletion breaks backwards compat more than expected | Medium | Medium | Greenfield policy covers this — we accepted retrain cost. Smoke tests validate pre/post equivalence in non-magnitude dimensions. | | Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. | -| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Feature branch stays unmerged. `feat/policy-quality` kept for rerun attempts. No pollution of main. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). | +| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Iterate on main with additional fix commits. Baseline tag `policy-quality-baseline` is rollback anchor. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). | | Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. | | Iteration cost exceeds ~7 hrs L40S (budgeted 5.5–6.5 for plan + 1 buffer) | Low | Low | Iteration budget hard-capped at 3 Phase-3 validation runs (3 hrs). If the 3rd still misses a mandatory gate, the spec escalates — no further validation runs on this branch. | | Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. |