diff --git a/docs/superpowers/specs/2026-04-21-policy-quality-design.md b/docs/superpowers/specs/2026-04-21-policy-quality-design.md index 93beb8ac1..37c5b51fe 100644 --- a/docs/superpowers/specs/2026-04-21-policy-quality-design.md +++ b/docs/superpowers/specs/2026-04-21-policy-quality-design.md @@ -53,42 +53,53 @@ Failing a gate does not trigger a revert. A follow-up spec (`docs/superpowers/sp ## 3. Architecture -Single branch. Phase-gated. Two parallel investigation tracks that merge into one convergence commit. +Feature-branch–isolated. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that merge into one convergence commit on the feature branch, validated, then merged to main via PR. ``` -main timeline: +feat/policy-quality branch: - [Phase 0] commit: measurement substrate + [Phase 0] commit: measurement substrate. Tagged policy-quality-baseline. ↓ (baseline run against this commit — reference metrics frozen) - [Phase 1] NO commits. Two parallel investigation tracks in local worktree - or throwaway branches. Findings documented in spec follow-up. + [Phase 1] NO commits on feat/*. Four parallel investigation tracks live + on throwaway branches wip/pq-track{1,2,3,4}, pushed to origin + for crash safety. ↓ - [Phase 2] commit: single convergence fix. Reward changes + magnitude sizer - fix + deleted sub-noise terms. All confirmed bugs fixed in one - commit. + [Phase 2] commit: single convergence fix on feat/policy-quality. + All confirmed bugs across all 4 tracks fix together. ↓ - [Phase 3] commit(s): validation tweaks from 6-fold run. Success-criteria - gate checked. Follow-up notes if partial. + [Phase 3] commit(s) on feat/*: validation tweaks from 6-fold run. + Success-criteria gate checked. + ↓ + + [Phase 4] Merge feat/policy-quality → main via PR. Only on all gates green. + If validation fails, feature branch stays, main untouched. ``` **Key architectural decisions:** +- **Feature branch for isolation.** Every change lives on `feat/policy-quality` until Phase 4 merge. `main` stays clean until validation passes. Revert is "don't merge." - **Measurement first.** No fix lands until its detection is codified in a smoke test. -- **No mid-investigation commits.** Phase 1 is exploratory; anything that lands pollutes the Phase 0 baseline. +- **No mid-investigation commits on feature branch.** Phase 1 is exploratory; only the convergence commit lands on `feat/policy-quality`. - **All confirmed findings fix together.** No ranking, no "most likely cause" — V7 discipline gates inclusion (measurement confirms a real bug), every confirmed bug ships in Phase 2. - **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production. - **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence. +- **Tagged baseline for rollback.** `git tag policy-quality-baseline` at Phase 0. If merged-to-main code ever regresses, `git reset --hard policy-quality-baseline` restores known state. -**Budget estimate:** ~3–5 days elapsed. +**Budget estimate:** ~4–6 days elapsed. - L40S validation runs: ~1 hour each, budget 3 runs = **3 hrs** (one after Phase 2, up to 2 more to iterate if partial) - Local smoke runs (RTX 3050): ~3 min each, budget unlimited -- Investigation HEALTH_DIAG runs during Phase 1 use the same Argo workflows, 20-epoch × 2-fold = ~12 min on L40S, budget 3 = ~30 min +- Investigation runs during Phase 1 — 4 tracks × 2-3 dedicated measurement runs each: + - Track 1 (magnitude): forced-exploration epoch experiment (~1 hr run) + baseline (~12 min) + per-hypothesis follow-up = ~1.5 hr + - Track 2 (reward audit): 1 instrumented 20-epoch run (~12 min) is enough to measure all 8 terms + - Track 3 (controllers): 1 instrumented run + 1 comparison run with selected controllers disabled = ~25 min + - Track 4 (exploration): 1 instrumented run (~12 min) + - Total investigation GPU: **~2.5 hrs** -Total GPU: **~3.5–4 hrs L40S** (not 6–10 — earlier estimate was conservative). +Total GPU: **~5.5–6.5 hrs L40S**. **Target GPU pool:** `ci-training-l40s` throughout. H100 only if L40S queue saturates (no expected architecture-specific differences). @@ -116,10 +127,25 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga - Terms below their class's threshold are triaged for DELETE in Phase 2. - Implementation detail: the test uses one training run with instrumentation, not 8 separate runs. +**`controller_activity.rs`** — Track 3 gate +- Train 20 epochs, record per-controller firing rate. +- Assert: no single controller (anti-LR, adaptive-tau, adaptive-gamma, grad-clip, CQL-alpha, cost-anneal) fires in > 50% of epochs. High firing rate = load-bearing → production risk. + +**`exploration_coverage.rs`** — Track 4 gate +- Train 20 epochs, record per-epoch magnitude-branch action entropy. +- Assert: entropy ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20. Collapse-speed check — if magnitude entropy hits 0 before epoch 20, something is quenching exploration. + **`multi_fold_convergence.rs`** — fast local variant of the full gate - 3 folds × 20 epochs (vs 6 × 50 on L40S). Runs ~5 min on RTX 3050. - Assert: all 3 folds complete without NaN, Best Sharpe > 0 on at least 2/3. +**`surrogate_noise_check.rs`** — false-positive detector +- Use the trained model from Phase 2 or Phase 3. +- On validation data, REPLACE model actions with uniformly random actions (same distribution shape as trained model). +- Run backtest with random actions. +- Assert: random-action Sharpe ≤ 0.5× trained-model Sharpe. If random actions produce comparable Sharpe, we have measurement/leakage bugs (look-ahead, data re-use, stale features) — not a real edge. +- Critical production-readiness gate. Permanent artifact — guards against future measurement regressions. + ### 4.2 HEALTH_DIAG extensions Add to the existing HEALTH_DIAG log line in `training_loop.rs`. All forward-only, no backward gradient, cost is one scalar readback per epoch per field. @@ -132,7 +158,16 @@ Track 1 fields (magnitude diagnosis): - `action_dist_quarter`, `action_dist_half`, `action_dist_full` — epoch-level usage fractions Track 2 fields (reward V7 audit): -- `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return +- `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return (see class-specific measurement in §4.1) + +Track 3 fields (controllers V7 audit): +- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch +- `controllers_firing_fraction` — aggregate: fraction of epochs where ANY controller fired non-trivially + +Track 4 fields (exploration V7 audit): +- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch +- `explore_entropy_direction` — same for direction branch (control variable) +- `noisy_net_sigma_mean` — average NoisyNets σ across weights ### 4.3 Phase 0 exit criteria @@ -146,31 +181,41 @@ Track 2 fields (reward V7 audit): **Branch discipline during Phase 1:** -- No commits land on `main`. -- Each track uses a throwaway local branch (`wip/policy-quality-track1`, `wip/policy-quality-track2`) that IS pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable. -- Track branches are never merged. At Phase 2, findings are re-applied by hand on a fresh `main` branch, and the throwaway branches are deleted. +- No commits land on `main` OR on `feat/policy-quality`. +- Each track uses a throwaway local branch (`wip/pq-track1-magnitude`, `wip/pq-track2-reward`, `wip/pq-track3-controllers`, `wip/pq-track4-exploration`) pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable. +- Track branches are never merged. At Phase 2, findings are re-applied on `feat/policy-quality` as a single commit, and the wip branches are deleted. -This gives us crash safety (branches pushed) + baseline cleanliness (nothing lands on main). +This gives us crash safety (branches pushed) + feature-branch cleanliness (nothing lands on `feat/policy-quality` until Phase 2). ### 5.1 Track 1 — Magnitude collapse diagnosis -Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** verdict based on the measurement signal. +Ten candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** verdict based on the measurement signal. H1–H5 are common RL-trading causes; H6–H10 are codebase-specific possibilities surfaced in spec review. | # | Hypothesis | Detection signal | Detection threshold | |---|---|---|---| -| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | During a forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | Ratio `mean(R_full) / std(R_full)` is ≤ 0.5× `mean(R_quarter) / std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. | +| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | Forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | `mean(R_full)/std(R_full)` is ≤ 0.5× `mean(R_quarter)/std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. | | H2 | Var[Q] conviction-sizer over-shrinking — `effective_max_pos *= 1/(1 + sqrt(Var[Q]))` permanently collapses the target toward Quarter. | `var_scale_mean` field | `var_scale_mean < 0.3` across ≥10/20 epochs | -| H3 | Kelly cap early-firing — Kelly stats (computed from Quarter-only history) cap at Quarter after 20+ trades. | `kelly_f_mean` field | `kelly_f_mean ≤ 0.25` across ≥10/20 epochs | +| H3 | **Kelly as hard multiplier is too aggressive for marginal edges** — Kelly formula `f = (bp − q)/b` gives small `f` when win rate × avg_win ≈ avg_loss (marginal edges, typical in trading). The code multiplies `effective_max_pos *= kelly_f`, shrinking the envelope before magnitude selection even sees it. Not a Kelly bug; a design question: should Kelly be a hard multiplier or a soft gate? | `kelly_f_mean` AND `avg_win / avg_loss` stats from training | `kelly_f_mean ≤ 0.25` AND `avg_win / avg_loss < 1.3` — confirms marginal-edge scenario where Kelly's low value is correct math but wrong policy tool. | | H4 | Magnitude-head gradient starvation — the per-action-branch gradient (*not* the per-loss-component IQN/CQL/C51/Ens split) to the magnitude head is much smaller than to the direction head. Magnitude weights stay near init. | `grad_ratio_mag_dir` field, measured as end-of-epoch average of per-step ‖∇ W[mag_head]‖ ÷ ‖∇ W[direction_head]‖ (pre-clip). | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs | | H5 | Bootstrap bias — initial random exploration anchored Quarter, epsilon decayed before the policy escaped. | Action entropy vs epsilon schedule | entropy < 0.3 before epoch 5 AND epsilon below 0.1 | +| H6 | Regime-adaptive trailing stop (wired 2026-04-21) closes Full-position trades prematurely. Full moves hit profit-retreat thresholds faster than Quarter moves because P&L magnitude is larger. | Per-magnitude trailing-stop firing rate, plus mean hold-time-at-exit per magnitude. | Trail firing rate on Full positions > 3× rate on Quarter positions, AND mean hold-time-at-exit on Full < 0.5× Quarter. | +| H7 | VSN (variable selection network) or NoisyNets-injected noise disproportionately masks/perturbs the magnitude-head's input features, so effective signal-to-noise is worse than direction head. | VSN mask distribution per branch (forward hook), NoisyNets σ per weight tensor. | Magnitude head mean-mask weight < 0.5× direction head, OR NoisyNets σ on w_b1fc > 2× other branch weights. | +| H8 | Target-network tau too slow for magnitude propagation — because magnitude distribution is concentrated at Quarter, target updates rarely propagate signals about Half/Full (never sampled enough), effective learning rate is lower for those bins. | Target-net weight drift per branch (tau × online_grad accumulated). | Magnitude-head target drift < 0.1× direction-head drift cumulative. | +| H9 | **Not a bug: data favors Quarter.** Most bars are low-volatility range; optimal sizing per bar IS Quarter. Removing magnitude branch entirely would be the correct design, not a fix. | Per-magnitude Sharpe during forced-exploration epoch (H1 measurement reused). | If all three magnitudes show similar Sharpe (no statistically significant difference), Quarter-collapse is correct behaviour and magnitude branch is dead code. | +| H10 | Entropy regularization + argmax tie-breaking: entropy bonus encourages uniform distribution across magnitudes; argmax at evaluation picks the first bin (Quarter by index) on any near-tie. Training-time exploration looks uniform but evaluation is quarter-biased. | Compare training-mode action distribution vs eval-mode action distribution. | Training-mode magnitude entropy > 0.8 (near-uniform) AND eval-mode `F_Quarter` > 90% (collapse at eval only). | **Fix sketch (if confirmed):** - H1 → rebalance reward gradient per magnitude (advantage weighting by bin, or per-bin loss scale) - H2 → clamp `var_scale ≥ 0.5` OR remove entirely if redundant with Kelly -- H3 → decouple Kelly from magnitude conditioning (use Kelly as position scaler, not magnitude quantizer) -- H4 → investigate why magnitude-head gradient is small. Candidates: magnitude-branch loss weight too low in the 4-branch advantage sum, magnitude head's `w_b1fc` dimensionality causing numerical suppression, VSN (variable selection network) masking magnitude features. Fix by rebalancing the 4-branch advantage weights OR adjusting initial weight scale for the magnitude head. (NOT the IQN/CQL/C51/Ens loss-component budget — that's orthogonal.) +- H3 → replace Kelly hard multiplier with a soft gate (e.g., `if kelly_f < 0.3 then magnitude ≤ Half`, keeping magnitude head free to choose); OR remove Kelly entirely and accept greater variance in exchange for proper action space exploration +- H4 → rebalance 4-branch advantage weights OR adjust initial weight scale for the magnitude head - H5 → extend exploration warmup for magnitude branch only +- H6 → feed per-magnitude scale into the trailing-stop distance (larger positions get proportionally wider stops), OR disable trailing stop entirely if H6 + H9 both confirmed +- H7 → mask VSN's effect on magnitude features; reduce NoisyNets σ on magnitude weights +- H8 → increase tau OR use per-branch target update rates (direction faster, magnitude slower with sample-weighted updates) +- H9 → delete magnitude branch, reduce action space from 108 to 36 (direction × order × urgency only). Massive simplification if this hypothesis confirms. +- H10 → break argmax ties by Q-value margin (no bin ordering bias); OR match training/eval distributions via stochastic eval rollouts ### 5.2 Track 2 — Reward V7 audit @@ -205,9 +250,84 @@ Every reward signal gets the V7 3-step treatment. | Contribution is meaningful AND invariant is not otherwise covered | KEEP | | Contribution appears meaningful but code is half-wired (e.g., forward only, no backward) | WIRE PROPERLY or DELETE based on backward-completion effort | -### 5.3 Phase 1 exit +### 5.3 Track 3 — Controllers V7 audit -Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item. +Every adaptive controller currently shaping training gets a V7 3-step treatment. + +**Controllers currently active:** + +| # | Controller | Current role | +|---|---|---| +| C1 | Anti-intuitive LR (asymmetric, wired 2026-04-21) | Fires every epoch after warmup; boosts LR on good signal, dampens on sustained bad | +| C2 | Adaptive tau (target-net update rate) | Adjusts based on Q-drift | +| C3 | Adaptive gamma (discount) | Curriculum: low gamma early, high later | +| C4 | Adaptive grad clip | Threshold adjusts to running grad-norm distribution | +| C5 | CQL alpha schedule | Pessimism weight vs online data confidence | +| C6 | Cost-anneal schedule (`cost_anneal_ptr`) | Tx-cost curriculum: 0 early, full later | +| C7 | Learning-rate scheduler (base) | Step decay or cosine | + +**Audit question per controller:** is this load-bearing (firing often, without it training breaks) or diagnostic (firing rarely, just catching edge cases)? A production-deployable policy should have controllers be diagnostic. + +**Decision matrix per controller:** + +| Evidence | Decision | +|---|---| +| Firing rate > 50% of epochs AND disabling it breaks training | LOAD-BEARING — keep, document the necessity, flag as production-readiness concern | +| Firing rate < 10% of epochs | DIAGNOSTIC — keep, no action | +| Firing rate 10–50% AND disabling it doesn't break training | CANDIDATE FOR DELETE — measure impact of removal, delete if within noise | +| Controller's trigger condition is symptom of upstream bug | ROOT-CAUSE FIX — fix upstream, controller becomes diagnostic | + +**Track 3 smoke test deliverable:** +```rust +// crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs +fn test_controllers_not_load_bearing() { + // Train 20 epochs. Record per-controller firing rate. + // Assert: no single controller fires in > 50% of epochs. + // High firing = policy can't hold on its own = production risk. +} +``` + +**HEALTH_DIAG extensions (Track 3):** +- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch, aggregated over window + +### 5.4 Track 4 — Exploration V7 audit + +**Exploration mechanisms currently active:** + +| # | Mechanism | Current role | +|---|---|---| +| E1 | Epsilon-greedy | Random action with probability ε, decays over training | +| E2 | NoisyNets | Injects parametric noise in fc layers for exploration | +| E3 | Count-based bonus (`count_bonus=true`) | Reward bonus for rarely-visited (state, action) pairs | +| E4 | Entropy regularization | Pushes policy toward uniform action dist | +| E5 | Counterfactual flip (R3) | Data-level exploration via mirroring (already in Track 2) | + +**Audit question per mechanism:** are these redundant with each other, and is the net exploration pressure at the action/magnitude level appropriate? + +**V7 audit template per mechanism:** +1. Identify signal — what exploration problem does this solve? +2. Check better-form — does another mechanism already provide this? +3. Measure — log magnitude-level action entropy, "novel state" count, per-mechanism contribution estimate +4. Decide — DELETE redundant, TUNE under-powered, KEEP covering unique problem + +**Track 4 smoke test deliverable:** +```rust +// crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs +fn test_exploration_covers_magnitude_space() { + // Train 20 epochs. Record per-epoch magnitude-level action entropy. + // Assert: entropy over magnitudes ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20. + // If entropy collapses too fast, magnitude space is under-explored. +} +``` + +**HEALTH_DIAG extensions (Track 4):** +- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch +- `explore_entropy_direction` — same for direction branch (control) +- `noisy_net_sigma_mean` — average NoisyNets σ across weights + +### 5.5 Phase 1 exit + +Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item. Four track outputs consolidate into one Phase 2 commit. --- @@ -226,18 +346,27 @@ Fixes confirmed by Track 1 (magnitude collapse): Deleted by Track 2 (reward V7 audit): - R (): sub-noise / better-form covered, deleted - - ... Kept by Track 2 (measurement confirmed real signal): - R: retained, +Fixes confirmed by Track 3 (controllers V7 audit): + - C (): + - Note if any controller remains LOAD-BEARING — flagged for sub-project B follow-up. + +Fixes confirmed by Track 4 (exploration V7 audit): + - E (): + Smoke tests added (Phase 0): - - magnitude_distribution — asserts ≥5% per bin - - reward_component_audit — asserts each active term ≥1% contrib - - multi_fold_convergence (fast local variant) + - magnitude_distribution (Track 1) + - reward_component_audit (Track 2) + - controller_activity (Track 3) + - exploration_coverage (Track 4) + - multi_fold_convergence (fast local variant of Phase 3 gate) + - surrogate_noise_check (false-positive detector) Validation gate (Phase 3): - - 6-fold × 50-epoch L40S run, success criteria A+B pending + - 6-fold × 50-epoch L40S run, success criteria A+B+surrogate-noise pending ``` **Rules:** @@ -267,8 +396,10 @@ Existing walk-forward config produces 6 folds on `test_data/futures-baseline`. | A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel | | **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log | | **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel | +| **Surrogate-noise (MANDATORY)** | Random-action Sharpe ≤ 0.5× trained-model Sharpe | `surrogate_noise_check.rs` smoke | +| Controller-activity (soft) | No controller fires in > 50% of epochs | HEALTH_DIAG `controller_fire_*` fields | -Criterion B gates MUST pass for the spec to close successfully. Criterion A gates are soft — if B passes but some A gate misses, the spec closes with a documented A follow-up. +Criterion B gates AND surrogate-noise check MUST pass for the spec to close successfully and merge to main (Phase 4). Criterion A gates are soft — if B and surrogate-noise pass but some A gate misses, the spec closes with a documented A follow-up. Controller-activity is soft — high firing rates go into a tech-debt follow-up note (sub-project B concern). ### 7.3 Partial-success handling @@ -284,8 +415,12 @@ Left in the tree after the spec closes: - `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs` - `crates/ml/src/trainers/dqn/smoke_tests/reward_component_audit.rs` +- `crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs` +- `crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs` - `crates/ml/src/trainers/dqn/smoke_tests/multi_fold_convergence.rs` -- HEALTH_DIAG extensions (Track 1 + Track 2 fields, all behavioral-code-path reusable) +- `crates/ml/src/trainers/dqn/smoke_tests/surrogate_noise_check.rs` +- HEALTH_DIAG extensions across all 4 tracks (all behavioral-code-path reusable) +- `git tag policy-quality-baseline` (Phase 0 anchor for rollback) These are the regression coverage for the invariants this spec establishes. @@ -306,10 +441,14 @@ These are the regression coverage for the invariants this spec establishes. | Risk | Likelihood | Impact | Mitigation | |---|---|---|---| -| Phase 1 investigation finds no confirmed bugs (all H1–H5 REJECTED) | Low | Medium | Track 2 reward audit still produces findings; if both tracks come up empty, upgrade spec to include controller audit (sub-project A scope creep). | -| Phase 2 fix passes smokes but validation (Phase 3) still misses action-dist gate | Medium | Medium | Partial-success path. Follow-up spec addresses remaining deltas. | +| Phase 1 finds all 10 Track-1 hypotheses REJECTED | Low | High | Track 2/3/4 almost certainly surface findings; if all 4 tracks come up empty, magnitude collapse might be **H9 (not a bug, data favors Quarter)**. Delete magnitude branch per H9 fix sketch — action-space simplification is still a production win. | +| H9 confirmed: magnitude branch deletion breaks backwards compat more than expected | Medium | Medium | Greenfield policy covers this — we accepted retrain cost. Smoke tests validate pre/post equivalence in non-magnitude dimensions. | +| Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. | +| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Feature branch stays unmerged. `feat/policy-quality` kept for rerun attempts. No pollution of main. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). | +| Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. | | Retraining cost runs beyond 10 hrs L40S | Low | Low | Iteration budget capped at 3 full validation runs. If still failing, triage with faster subset runs. | | Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. | +| Controller-audit (Track 3) finds multiple load-bearing controllers that can't be removed in this spec | Medium | Medium | Document as tech-debt, flag for sub-project B. Spec can still close if policy is profitable. But "load-bearing controllers = not production-deployable" remains a known concern. | ---