gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(policy-quality): main-branch workflow — cut feat/wip branches

Browse Source 
User preference: everything lands on main, ~500 LOC is small enough
that feature-branch isolation isn't needed.

Spec changes:
  * §2.3 outcome paths: drop "unmerged branch" vocabulary
  * §3 architecture: main-branch timeline, no feat/, no wip/
  * §4 Phase 0: commits on main, tag policy-quality-baseline at end
  * §5 Phase 1: experiments are worktree edits reverted after, only
    triage docs commit (to main)
  * §6 Phase 2: commits land on main, author chooses granularity,
    each leaves smokes green
  * §7.3 outcome handling: iterate on main, baseline tag for rollback
  * §7.4 closing: tag policy-quality-v1, update status

Plan changes:
  * Task 0.1: verify-starting-state (not branch creation)
  * Task 1.x: temp-edit-then-revert (not wip branches)
  * Task 1.5: triage docs commit directly to main
  * Phase 4: tag + close (not merge + cleanup)
  * Rollback: one git reset --hard command
This commit is contained in:
jgrusewski
2026-04-21 21:06:33 +02:00
parent 1456094a5a
commit 944fecd913
2 changed files with 150 additions and 233 deletions
Download Patch File Download Diff File Expand all files Collapse all files

292
docs/superpowers/plans/2026-04-21-policy-quality.md
View File

@@ -4,7 +4,7 @@
**Goal:** Execute the 4-phase V7 audit specified in `docs/superpowers/specs/2026-04-21-policy-quality-design.md` — build measurement substrate, run parallel investigation tracks, land all confirmed fixes in a single convergence commit, validate on 6-fold × 50-epoch L40S, merge to main.
**Architecture:** Feature branch `feat/policy-quality` isolates work from main. Phase 0 commit + Phase 2 commit + Phase 3 fixup commits on the feature branch. Merge to main via `--no-ff` PR only after MANDATORY gates (B-Sharpe, B-WinRate, surrogate-noise) pass. Failures don't pollute main.
**Architecture:** All commits land on `main`. Tagged `policy-quality-baseline` after Phase 0 as rollback anchor. Phase 2 fixes commit incrementally — each leaves main working. Closing tag `policy-quality-v1` on the final validated commit.
**Tech Stack:** Rust (workspace crates: ml, ml-dqn, ml-core), CUDA (cuBLAS, custom kernels via nvcc), Argo Workflows on Scaleway Kapsule, L40S GPU (sm_89), fxcache binary feature cache, `#[ignore]` smoke tests gated on `FOXHUNT_TEST_DATA`.
@@ -73,7 +73,7 @@ Per-finding. Recorded during Phase 2 planning commit.
**Exit criteria:** All 6 new smoke tests compile, run, and produce readable output. HEALTH_DIAG fields appear in training logs with non-zero values.
**Phase 0 commits (in order):**
1. Create feature branch + tag baseline
1. Verify starting state
2. HEALTH_DIAG field scaffolding (host-side, zero-returning stubs)
3. Track 1 HEALTH_DIAG wiring (GPU-side readbacks for H1–H10 metrics)
4. Track 2 HEALTH_DIAG wiring (reward-term contribution)
@@ -84,41 +84,25 @@ Per-finding. Recorded during Phase 2 planning commit.
---
### Task 0.1: Create feature branch and baseline tag
### Task 0.1: Verify starting state
**Files:**
- Modify: git only
- git only
- [ ] **Step 1: Verify main is at expected HEAD**
Run: `git fetch origin && git log --oneline -1 origin/main`
Expected: output shows `ad677466c design(policy-quality): revision 4 — functional gaps` or later.
- [ ] **Step 2: Create feature branch from main**
- [ ] **Step 1: Verify on main, up to date**
Run:
```bash
git checkout main
git pull --ff-only
git checkout -b feat/policy-quality
git pull --ff-only origin main
git log -1 --oneline
```
Expected: `Switched to a new branch 'feat/policy-quality'`
Expected: clean working tree, at latest main (spec/plan commits present).
- [ ] **Step 3: Push branch to origin with upstream tracking**
- [ ] **Step 2: Note the baseline SHA**
Run: `git push -u origin feat/policy-quality`
Expected: `branch 'feat/policy-quality' set up to track 'origin/feat/policy-quality'.`
- [ ] **Step 4: Create baseline tag (will move to Phase-0-final commit at end of Phase 0)**
Run:
```bash
# Tag the CURRENT HEAD — we'll move this tag after Phase 0 completes.
# For now it's an anchor reference.
git tag -f policy-quality-baseline HEAD
git push -f origin policy-quality-baseline
```
Expected: tag pushed, `git tag -l policy-quality-baseline` lists it.
Run: `git rev-parse HEAD | tee /tmp/policy-quality-pre-phase0-sha`
Expected: one SHA. Kept for reference — we'll tag the Phase-0-final commit (not this SHA) as `policy-quality-baseline` at Task 0.17.
---
@@ -1265,53 +1249,30 @@ git commit -m "docs(policy-quality): Phase 0 baseline metrics (pre-audit referen
git push
```
- [ ] **Step 4: Move the `policy-quality-baseline` tag to the Phase 0 head**
- [ ] **Step 4: Tag the final Phase-0 commit as `policy-quality-baseline`**
```bash
git tag -f policy-quality-baseline HEAD
git push -f origin policy-quality-baseline
git tag -a policy-quality-baseline -m "Phase 0 complete — instrumentation in, pre-fix reference"
git push origin policy-quality-baseline
```
This tag is the rollback anchor for Phases 2+. If something breaks on main, `git reset --hard policy-quality-baseline` restores this known-good state.
---
## Phase 1 — Parallel Investigation Tracks
## Phase 1 — Investigation Tracks
**Goal:** run each track's HEALTH_DIAG-instrumented training, collect evidence, classify each hypothesis/term/controller/mechanism as CONFIRMED or REJECTED.
**Exit criteria:** Triage document per track committed to the corresponding `wip/pq-track*` branch. Track outputs merged into a single Phase 2 planning document.
**Exit criteria:** 4 triage documents committed to main. Phase-2 implementation plan written from consolidated findings.
**Branch discipline:** each track lives on `wip/pq-track{1,2,3,4}-*`. Nothing lands on `feat/policy-quality` during Phase 1.
---
### Task 1.0: Create investigation branches
- [ ] **Step 1: From `feat/policy-quality`, create 4 wip branches**
```bash
git checkout feat/policy-quality
for track in 1-magnitude 2-reward 3-controllers 4-exploration; do
git checkout -b wip/pq-track${track}
git push -u origin wip/pq-track${track}
done
git checkout feat/policy-quality
```
Each wip branch is a parallel universe for investigation. Changes here don't land on `feat/policy-quality`.
**Discipline:** some hypotheses need temporary code edits (e.g., force epsilon=1.0 for H1). Edit the file, run the experiment, then `git checkout -- <file>` to revert. Don't commit experimental code. Only triage docs commit.
---
### Task 1.1: Track 1 — Magnitude investigation
**Branch:** `wip/pq-track1-magnitude`
- [ ] **Step 1: Checkout track branch**
```bash
git checkout wip/pq-track1-magnitude
```
- [ ] **Step 2: Baseline run — inherits Phase 0 instrumentation**
- [ ] **Step 1: Baseline run — inherits Phase 0 instrumentation**
Already have HEALTH_DIAG from Phase 0. Run 20-epoch × 2-fold on L40S:
@@ -1324,18 +1285,25 @@ Wait for completion, dump logs:
argo logs -n foxhunt <workflow-name> > /tmp/track1-baseline.log
```
- [ ] **Step 3: Forced-exploration experiment (for H1 detection)**
- [ ] **Step 2: Forced-exploration experiment (for H1 detection)**
H1 requires epsilon=1.0 for 1 epoch at mid-training. Edit `training_loop.rs` temporarily to force epsilon=1.0 at epoch 10, re-run workflow, dump logs as `/tmp/track1-forced-explore.log`. Do NOT commit the temporary change.
H1 requires epsilon=1.0 for 1 epoch at mid-training.
- [ ] **Step 4: Classify each hypothesis**
1. Edit `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — force `epsilon = 1.0` when `epoch == 10`.
2. Run the workflow:
```bash
./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline
argo logs -n foxhunt <workflow> > /tmp/track1-forced-explore.log
```
3. Revert the edit: `git checkout -- crates/ml/src/trainers/dqn/trainer/training_loop.rs`
- [ ] **Step 3: Classify each hypothesis**
For each of H1–H10, check the detection signal against the threshold per spec §5.1. Record CONFIRMED or REJECTED with evidence in `docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md`:
```markdown
# Track 1 — Magnitude Collapse Triage
**Branch:** wip/pq-track1-magnitude
**Log refs:** /tmp/track1-baseline.log, /tmp/track1-forced-explore.log
## H1 — Reward gradient asymmetry
@@ -1348,7 +1316,8 @@ For each of H1–H10, check the detection signal against the threshold per spec
(...repeat for H2–H10)
```
Commit to the wip branch:
- [ ] **Step 4: Commit triage doc to main**
```bash
git add docs/superpowers/specs/2026-04-21-policy-quality-track1-triage.md
git commit -m "triage(track1): magnitude hypotheses H1-H10 verdicts with evidence"
@@ -1359,105 +1328,108 @@ git push
### Task 1.2: Track 2 — Reward V7 audit
**Branch:** `wip/pq-track2-reward`
- [ ] **Step 1: Checkout + baseline run**
- [ ] **Step 1: Baseline run**
```bash
git checkout wip/pq-track2-reward
./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline
argo logs -n foxhunt <workflow> > /tmp/track2-baseline.log
```
Phase 0 reward-contrib HEALTH_DIAG fields already capture per-term signals.
- [ ] **Step 2: For each of R1–R8, apply the V7 audit template**
Per spec §5.2:
1. Identify signal (git log, grep for the term, read its introducing commit)
2. Check better-form (does another term cover the same invariant?)
3. Measure (reward_contrib_* field from HEALTH_DIAG over 20 epochs)
3. Measure (`reward_contrib_*` field from `/tmp/track2-baseline.log`)
4. Decide per the decision matrix
- [ ] **Step 3: Record triage**
- [ ] **Step 3: Record triage + commit**
Write `docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md`:
Write `docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md` (one section per R1–R8) and commit to main:
```markdown
# Track 2 — Reward V7 Audit Triage
(one section per R1–R8 with identify / better-form / measured / decision)
```bash
git add docs/superpowers/specs/2026-04-21-policy-quality-track2-triage.md
git commit -m "triage(track2): reward V7 audit R1-R8 verdicts"
git push
```
Commit to wip branch.
---
### Task 1.3: Track 3 — Controllers V7 audit
**Branch:** `wip/pq-track3-controllers`
- [ ] **Step 1: Use the Phase 0 HEALTH_DIAG controller-fire-rate data (already captured)**
- [ ] **Step 1: Checkout + baseline run**
Controller fire rates appear in every HEALTH_DIAG line. Extract from `/tmp/track2-baseline.log` or run another 20-epoch if needed.
Same as 1.2 but focused on controller fire rates.
- [ ] **Step 2: Disable-and-measure experiments**
- [ ] **Step 2: Disable-and-measure experiment**
For candidates flagged "CANDIDATE FOR DELETE" per spec §5.3 (10–50% fire rate), disable each temporarily and compare outcomes:
For candidates flagged as "CANDIDATE FOR DELETE" (10–50% fire rate), temporarily disable each one (gate with a flag), retrain, compare outcomes. Record delta.
1. Edit `training_loop.rs` — change the controller's if-block to `if false`.
2. `./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 20 --baseline`
3. Record Best Sharpe + grad stability vs baseline in the triage doc.
4. Revert: `git checkout -- crates/ml/src/trainers/dqn/trainer/training_loop.rs`
5. Repeat for next candidate.
- [ ] **Step 3: Triage document**
- [ ] **Step 3: Triage document + commit**
Write `docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md`, one section per C1–C7. Commit.
Write `docs/superpowers/specs/2026-04-21-policy-quality-track3-triage.md`, one section per C1–C7. Commit to main.
---
### Task 1.4: Track 4 — Exploration V7 audit
**Branch:** `wip/pq-track4-exploration`
- [ ] **Step 1: Use Phase 0 HEALTH_DIAG data**
- [ ] **Step 1: Baseline run (from Phase 0)**
HEALTH_DIAG already captures `explore_entropy_*` and `noisy_net_sigma_mean`. No additional instrumentation needed.
`explore_entropy_*` and `noisy_sigma_mean` are already captured. No additional instrumentation needed for E1/E2/E4.
- [ ] **Step 2: For each E1–E5, apply V7 template**
Count-based bonus (E3) and entropy regularization (E4) may be redundant with NoisyNets (E2). Measure each mechanism's contribution to per-epoch action-entropy. Run with each mechanism disabled in turn (5 short runs).
Count-based bonus (E3) and entropy regularization (E4) may be redundant with NoisyNets (E2). For each mechanism:
1. Edit its enable-block to `if false` in training_loop.rs
2. Run 20-epoch
3. Record impact on magnitude entropy
4. `git checkout -- <file>`, move to next
- [ ] **Step 3: Triage document**
- [ ] **Step 3: Triage document + commit**
Write `docs/superpowers/specs/2026-04-21-policy-quality-track4-triage.md`. Commit.
Write `docs/superpowers/specs/2026-04-21-policy-quality-track4-triage.md`. Commit to main.
---
### Task 1.5: Consolidate 4 triage documents into Phase 2 planning
### Task 1.5: Write Phase 2 implementation plan
- [ ] **Step 1: Back on `feat/policy-quality`**
- [ ] **Step 1: Consolidate findings**
All 4 triage docs are on main. Skim each, compile a list of CONFIRMED findings.
- [ ] **Step 2: Invoke writing-plans against the consolidated triage**
```bash
git checkout feat/policy-quality
# Use superpowers:writing-plans with input:
# "Given the 4 triage docs at
# docs/superpowers/specs/2026-04-21-policy-quality-track{1,2,3,4}-triage.md
# produce a Phase 2 implementation plan with concrete TDD tasks per CONFIRMED
# finding. Target: docs/superpowers/plans/2026-04-21-policy-quality-phase2.md"
```
- [ ] **Step 2: Cherry-pick the four triage documents**
- [ ] **Step 3: Commit the Phase 2 plan**
```bash
for track in 1 2 3 4; do
git checkout wip/pq-track${track}-* -- docs/superpowers/specs/2026-04-21-policy-quality-track${track}-triage.md
done
git add docs/superpowers/specs/2026-04-21-policy-quality-track*-triage.md
git commit -m "triage(policy-quality): consolidate 4-track triage docs to feat branch"
git add docs/superpowers/plans/2026-04-21-policy-quality-phase2.md
git commit -m "plan(policy-quality): Phase 2 implementation plan from triage findings"
git push
```
- [ ] **Step 3: Write Phase 2 implementation plan**
Based on the triage docs, write `docs/superpowers/plans/2026-04-21-policy-quality-phase2.md` with concrete TDD tasks for each CONFIRMED finding. This is a separate planning pass — invoke the `writing-plans` skill against the consolidated triage.
Commit the Phase 2 plan.
---
## Phase 2 — Convergence Fix
**Goal:** implement all fixes identified in Phase 1 triage, in a single commit on `feat/policy-quality`. Every new smoke test (Track 1–4) passes at its production threshold, not just the relaxed smoke-gate threshold.
**Goal:** implement all fixes identified in Phase 1 triage on main. Each commit leaves main working + smokes green.
**Exit criteria:** all 6 Phase 0 smokes pass; existing smokes (td_propagation, test_50_epoch_convergence) still pass.
**Exit criteria:** all 6 Phase 0 smokes pass at their PRODUCTION thresholds (tighten from Phase 0's relaxed smoke thresholds). Existing smokes (`td_propagation`, `test_50_epoch_convergence`) still pass.
**Template for Phase 2 tasks (filled in by the Phase 2 plan written in Task 1.5):**
@@ -1465,14 +1437,10 @@ For each CONFIRMED finding in any of the 4 triages:
1. Write test asserting the fixed behaviour (e.g., magnitude bin usage at production threshold)
2. Implement the fix per the spec's fix sketch
3. Run all 6 Phase 0 smokes locally — expect all pass
4. Amend into the single Phase 2 convergence commit
4. Commit with message referencing the triage finding (e.g., `fix(dqn): H2 — clamp var_scale ≥ 0.5 per track1 triage`)
5. Push
Single final commit message:
```
fix(dqn): policy-quality V7 audit — <N> bugs, <M> dead terms removed
(full enumeration per spec §6 commit structure)
```
Commits can be one-per-finding (preferred — easier to review) or grouped by track. Author's choice.
---
@@ -1482,7 +1450,7 @@ fix(dqn): policy-quality V7 audit — <N> bugs, <M> dead terms removed
### Task 3.1: Run validation workflow
- [ ] **Step 1: Ensure `feat/policy-quality` HEAD is pushed and committed by Phase 2**
- [ ] **Step 1: Ensure main HEAD has Phase 2 fixes committed and pushed**
Run: `git status && git log -1 --oneline`
Expected: clean, last commit is the Phase 2 convergence.
@@ -1571,7 +1539,7 @@ Expected: pass — trained pooled Sharpe exceeds 95th %ile of N=30 surrogates.
Commit `phase3-results.md` and proceed to merge.
- [ ] **Step 2b: If any mandatory fails → iterate on `feat/policy-quality`**
- [ ] **Step 2b: If any mandatory fails → iterate on main**
Up to 2 additional iterations total (max 3 Phase-3 validation runs). Each iteration:
- Diagnose from log which gate failed and why
@@ -1583,118 +1551,68 @@ Up to 2 additional iterations total (max 3 Phase-3 validation runs). Each iterat
---
## Phase 4 — Merge
## Phase 4 — Close
**Goal:** merge `feat/policy-quality` to main, tag release, clean up branches.
**Goal:** tag `policy-quality-v1` on the final validated main commit, update the spec status, close.
**Prerequisites:** Phase 3 all MANDATORY gates pass.
### Task 4.1: Final pre-merge verification
### Task 4.1: Tag the validated commit
- [ ] **Step 1: Fast-forward main on local**
- [ ] **Step 1: Confirm on main at the post-Phase-3 HEAD**
```bash
git checkout main
git pull --ff-only origin main
git log -1 --oneline
```
- [ ] **Step 2: Confirm feat branch is up to date**
- [ ] **Step 2: Tag**
```bash
git checkout feat/policy-quality
git log origin/main..HEAD --oneline
```
git tag -a policy-quality-v1 -m "Policy quality V7 audit complete — sub-project A closed.
Expected: all Phase 0 through Phase 3 commits listed, nothing unexpected.
### Task 4.2: Merge via `--no-ff`
- [ ] **Step 1: Return to main, merge feat branch**
```bash
git checkout main
git merge --no-ff feat/policy-quality -m "Merge feat/policy-quality: policy-quality V7 audit
Sub-project A of production-readiness roadmap complete.
All MANDATORY gates passed in Phase 3 validation:
- B-Sharpe: Best val Sharpe > 10 on <N>/6 folds
- B-WinRate: Val WinRate > 55% on <N>/6 folds
- Surrogate-noise: trained pooled Sharpe beyond 95th %ile of 30 random surrogates
Phase 2 convergence commit: <sha>
Phase 3 validation results: docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md
"
```
- [ ] **Step 2: Push main**
```bash
git push origin main
```
### Task 4.3: Tag + cleanup
- [ ] **Step 1: Tag release**
```bash
MERGE_COMMIT=$(git rev-parse HEAD)
git tag -a policy-quality-v1 -m "Policy quality V7 audit complete — sub-project A closed" "$MERGE_COMMIT"
Phase 3 results: docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md"
git push origin policy-quality-v1
```
- [ ] **Step 2: Delete local feature branch**
### Task 4.2: Update spec status + roadmap
```bash
git branch -d feat/policy-quality
git push origin --delete feat/policy-quality
```
- [ ] **Step 1: Add CLOSED header to the spec**
- [ ] **Step 3: Keep wip/pq-track* branches for 30 days as audit trail**
Set a calendar reminder (or open a follow-up task) to run in 30 days:
```bash
for track in 1-magnitude 2-reward 3-controllers 4-exploration; do
git push origin --delete wip/pq-track${track}
done
```
### Task 4.4: Close spec + update roadmap
- [ ] **Step 1: Append closure note to the spec**
Add at top of `docs/superpowers/specs/2026-04-21-policy-quality-design.md`:
At top of `docs/superpowers/specs/2026-04-21-policy-quality-design.md`, insert:
```markdown
**Status (2026-MM-DD): CLOSED — merged to main as `policy-quality-v1`.**
**Phase 3 results:** see `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md`
**Status (2026-MM-DD): CLOSED — tagged `policy-quality-v1`.**
**Phase 3 results:** `docs/superpowers/specs/2026-04-21-policy-quality-phase3-results.md`
```
- [ ] **Step 2: Update production-readiness roadmap**
If sub-project roadmap doc exists, check off sub-project A. Note unblocking of sub-projects B, C.
- [ ] **Step 3: Commit + push**
- [ ] **Step 2: Commit + push**
```bash
git add docs/superpowers/specs/2026-04-21-policy-quality-design.md
git commit -m "docs(policy-quality): mark sub-project A closed, merged as policy-quality-v1"
git commit -m "docs(policy-quality): mark sub-project A closed, tagged policy-quality-v1"
git push
```
Sub-projects B, C, D (validation rigor, paper trading, production ops) are now unblocked.
---
## Rollback Procedure (emergency)
If `policy-quality-v1` causes production issues:
If a commit on main breaks something that wasn't caught by smokes:
```bash
git checkout main
git reset --hard policy-quality-baseline
git push -f origin main # only if team coordination allows
git push -f origin main # only with team coordination
```
The baseline tag points at the Phase 0 commit — the last known-good state before any Phase 2 behavioural change. Recovery is one command.
The baseline tag = last Phase 0 commit = pre-fix reference state. Recovery is one command.
---

91
docs/superpowers/specs/2026-04-21-policy-quality-design.md
View File

@@ -34,7 +34,7 @@ Paper trading integration, canary deployment, hyperopt re-runs, and architecture
## 2. Success criteria
The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the `feat/policy-quality` branch is merged to main via PR.
The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the final main commit is tagged `policy-quality-v1`.
### 2.1 Mandatory gates (all must pass to merge)
@@ -51,52 +51,53 @@ The spec closes when a single L40S validation run (6-fold × 50-epoch, the exist
### 2.3 Outcome paths
- **All mandatory + all soft pass** → merge to main, close spec clean.
- **All mandatory pass, some soft miss** → merge to main, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas.
- **Any mandatory miss** → `feat/policy-quality` stays unmerged. Up to 2 more iterations on the feature branch (total 3 Phase-3 validation runs). If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with:
- **All mandatory + all soft pass** → tag `policy-quality-v1`, close spec clean.
- **All mandatory pass, some soft miss** → tag `policy-quality-v1`, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. Does not block sub-project B.
- **Any mandatory miss** → iterate on main (up to 2 more iterations, total 3 Phase-3 validation runs). Use `policy-quality-baseline` tag to revert if an iteration makes things worse. If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with:
- Summary of the 3 failed runs' evidence (which gates, by how much, which hypotheses correlate with failure)
- Explicit diagnosis: policy/reward/state bottleneck? architecture bottleneck? data bottleneck?
- Decision: continue on `feat/policy-quality` with new hypotheses, OR fork a new branch `feat/policy-quality-v2` with a different angle (e.g., reward redesign, architecture change, alternative algorithm PPO/SAC)
- Triage spec is a full brainstorming cycle (not a follow-up note) — restarts the skill flow.
- Decision: continue on main with new hypotheses, OR scope a different angle (reward redesign, architecture change, alternative algorithm PPO/SAC) — the triage spec is a full brainstorming cycle that restarts the skill flow.
---
## 3. Architecture
Feature-branch–isolated. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that merge into one convergence commit on the feature branch, validated, then merged to main via PR.
Main-branch only. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that converge into the Phase-2 fix commit set. Tagged baseline for rollback.
```
feat/policy-quality branch:
main branch:
[Phase 0] commit: measurement substrate. Tagged policy-quality-baseline.
↓ (baseline run against this commit — reference metrics frozen)
[Phase 0] commit(s): measurement substrate. Final commit tagged
↓ policy-quality-baseline. Baseline metrics captured here.
[Phase 1] NO commits on feat/*. Four parallel investigation tracks live
on throwaway branches wip/pq-track{1,2,3,4}, pushed to origin
for crash safety.
[Phase 1] Investigation. Temporary experiments in local worktree /
git-stash — not committed (experiments are throwaway). Final
output is triage documents committed to main.
↓
[Phase 2] commit: single convergence fix on feat/policy-quality.
All confirmed bugs across all 4 tracks fix together.
[Phase 2] commit(s): convergence fixes. Can be one commit or several
per confirmed finding — main is our working branch. Total
~500 LOC expected; commit granularity is author's choice.
↓
[Phase 3] commit(s) on feat/*: validation tweaks from 6-fold run.
[Phase 3] commit(s): validation iteration tweaks from 6-fold L40S run.
Success-criteria gate checked.
↓
[Phase 4] Merge feat/policy-quality → main via PR. Only on all gates green.
If validation fails, feature branch stays, main untouched.
[CLOSE] Tag policy-quality-v1 on the final validated commit.
If validation fails after 3 iterations, open triage spec
per §2.3.
```
**Key architectural decisions:**
- **Feature branch for isolation.** Every change lives on `feat/policy-quality` until Phase 4 merge. `main` stays clean until validation passes. Revert is "don't merge."
- **Main-branch workflow.** All commits land on main. ~500 LOC total is small enough that feature-branch isolation isn't needed.
- **Measurement first.** No fix lands until its detection is codified in a smoke test.
- **No mid-investigation commits on feature branch.** Phase 1 is exploratory; only the convergence commit lands on `feat/policy-quality`.
- **All confirmed findings fix together.** No ranking, no "most likely cause" — V7 discipline gates inclusion (measurement confirms a real bug), every confirmed bug ships in Phase 2.
- **Phase 1 investigation stays local.** Experimental instrumentation lives in the worktree during measurement runs, never committed. Only the resulting triage documents commit to main.
- **Fixes can be multiple commits.** The spec's earlier "single convergence commit" rule is relaxed — each confirmed finding can be its own commit if that's easier to review. The discipline that remains: all fixes land before Phase 3 validation runs; each commit leaves smokes green.
- **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production.
- **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence.
- **Tagged baseline for rollback.** `git tag policy-quality-baseline` at Phase 0. If merged-to-main code ever regresses, `git reset --hard policy-quality-baseline` restores known state.
- **Tagged baseline for rollback.** `git tag policy-quality-baseline` after Phase 0 completes. If Phase 2+ breaks main, `git reset --hard policy-quality-baseline` restores known state. Tag `policy-quality-v1` marks the validated close.
**Budget estimate:** ~4–6 days elapsed.
@@ -117,9 +118,9 @@ Total GPU: **~5.5–6.5 hrs L40S**.
## 4. Phase 0 — Measurement substrate
Single commit on `feat/policy-quality` (branch created from current `main` HEAD). Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds).
One or more commits on `main`. Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds).
Tag this commit `policy-quality-baseline` for rollback anchor.
After Phase 0 completes, tag the final Phase-0 commit `policy-quality-baseline` for rollback anchor.
### 4.1 New smoke tests (ignored, run manually)
@@ -197,7 +198,8 @@ Track 4 fields (exploration V7 audit):
- All new smoke tests compile and produce output. The Phase 0 smokes use **relaxed thresholds** (e.g. magnitude ≥5% not 15%) so they pass current code; they will tighten in later commits or their production equivalent lives in the Phase 3 validation gate.
- HEALTH_DIAG fields appear in training logs with non-zero values across a baseline 20-epoch run.
- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to `feat/policy-quality` — audit trail, not investigation-only.
- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to main — audit trail.
- `git tag policy-quality-baseline` on the final Phase-0 commit + `git push origin policy-quality-baseline`.
---
@@ -205,11 +207,11 @@ Track 4 fields (exploration V7 audit):
**Branch discipline during Phase 1:**
- No commits land on `main` OR on `feat/policy-quality`.
- Each track uses a throwaway local branch (`wip/pq-track1-magnitude`, `wip/pq-track2-reward`, `wip/pq-track3-controllers`, `wip/pq-track4-exploration`) pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable.
- Track branches are never merged. At Phase 2, findings are re-applied on `feat/policy-quality` as a single commit, and the wip branches are deleted.
- No **code** commits land on `main` during Phase 1 — the temporary instrumentation (e.g., forced-exploration epsilon=1.0 experiment, controller-disable experiments) is experimental and would pollute history.
- **Triage documents DO commit to `main`** as they're completed. Each track's triage doc is a permanent artifact.
- Experimental code changes live in the worktree (uncommitted) or in local `git stash` during a measurement run; discard after the triage doc is written.
This gives us crash safety (branches pushed) + feature-branch cleanliness (nothing lands on `feat/policy-quality` until Phase 2).
This is the pragma: instrumentation commits from Phase 0 are kept (useful), investigation code is throwaway, triage output is committed.
### 5.1 Track 1 — Magnitude collapse diagnosis
@@ -363,9 +365,9 @@ Tracks can reach conflicting conclusions — e.g. Track 1-H6 might say "fix trai
---
## 6. Phase 2 — Convergence commit
## 6. Phase 2 — Convergence fix commits
Single commit on `feat/policy-quality`. Lands all Phase 1 findings together. Phase-4 merge to main happens only after Phase-3 validation passes — see §7.
Commits on `main` — one or several, author's choice. All Phase 1 findings fixed before Phase 3 validation begins. Each commit leaves smokes green.
**Commit structure:**
@@ -403,9 +405,10 @@ Validation gate (Phase 3):
**Rules:**
- **Single commit on `feat/policy-quality`.** Any bug discovered during Phase 2 integration testing goes into the same commit via amend.
- **All smoke tests green before push:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`).
- **Commits land on main.** Author chooses granularity (one per finding, or grouped). Each commit leaves main in a working state.
- **All smoke tests green before push on every commit:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`).
- **No config-file-only changes.** Every behavioral change has source code attached.
- **Commit messages reference the triage finding** (e.g., "fix(dqn): H2 — clamp var_scale ≥ 0.5 per track1 triage"). Makes git log readable as the audit trail.
---
@@ -439,23 +442,19 @@ Criterion B gates AND surrogate-noise check MUST pass for the spec to close succ
See §2.3 for the outcome paths. Concretely:
- **Missed mandatory gate** → `feat/policy-quality` stays unmerged. Iterate up to 2 more validation runs with additional commits on the feature branch. If 3 runs total still miss a mandatory gate, escalate out of this spec.
- **Missed soft gate only** → merge to main via `--no-ff` PR; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope.
- **Missed mandatory gate** → iterate up to 2 more validation runs with additional commits on main. If 3 runs total still miss a mandatory gate, escalate out of this spec (triage doc per §2.3).
- **Missed soft gate only** → tag `policy-quality-v1`; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope.
### 7.4 Phase 4 — Merge to main
### 7.4 Closing the spec
When mandatory gates pass:
```bash
# After mandatory gates pass:
git checkout main
git merge --no-ff feat/policy-quality # preserves branch identity in history
git push origin main
git tag policy-quality-v1 <merge-commit> # named checkpoint
git branch -d feat/policy-quality # local cleanup
git push origin --delete feat/policy-quality
# Leave wip/pq-track{1..4} branches for 30 days as investigation audit trail, then delete.
git tag -a policy-quality-v1 -m "Policy quality V7 audit — sub-project A closed"
git push origin policy-quality-v1
```
`--no-ff` is deliberate — the merge-commit documents "this is where the 4-track V7 audit landed" in the history, even if the branch is linear.
Update the top of this design doc with `Status: CLOSED`, reference the Phase-3 results doc, and mark sub-project A complete in any production-readiness tracking.
### 7.5 Permanent artifacts (survive spec closure)
@@ -492,7 +491,7 @@ These are the regression coverage for the invariants this spec establishes.
| Phase 1 finds all 10 Track-1 hypotheses REJECTED | Low | High | Track 2/3/4 almost certainly surface findings; if all 4 tracks come up empty, magnitude collapse might be **H9 (not a bug, data favors Quarter)**. Delete magnitude branch per H9 fix sketch — action-space simplification is still a production win. |
| H9 confirmed: magnitude branch deletion breaks backwards compat more than expected | Medium | Medium | Greenfield policy covers this — we accepted retrain cost. Smoke tests validate pre/post equivalence in non-magnitude dimensions. |
| Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. |
| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Feature branch stays unmerged. `feat/policy-quality` kept for rerun attempts. No pollution of main. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). |
| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Iterate on main with additional fix commits. Baseline tag `policy-quality-baseline` is rollback anchor. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). |
| Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. |
| Iteration cost exceeds ~7 hrs L40S (budgeted 5.5–6.5 for plan + 1 buffer) | Low | Low | Iteration budget hard-capped at 3 Phase-3 validation runs (3 hrs). If the 3rd still misses a mandatory gate, the spec escalates — no further validation runs on this branch. |
| Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. |