# Policy Quality — V7 Audit and Magnitude Collapse Fix

**Spec type:** sub-project A of production-readiness roadmap
**Status:** design approved, ready for implementation plan
**Target:** Foxhunt DQN policy quality — from "Fold 0 converges" to "multi-fold consistent with meaningful trade volume"
**Preconditions:** today's stability fixes are in main (v_range clamp, fold-boundary EMA reset, asymmetric anti-LR, graph-capture dtoh fix, sccache)

---

## 1. Context

### 1.1 Observed policy pathologies

From `train-sqnqh` (20-epoch × 2-fold L40S, 2026-04-21):

- **Fold 0** converges: Best val Sharpe 29.39 annualized, val WinRate 64–71%, 19–25 trades/val window.
- **Fold 1** now converges (no NaN, no CUDA_ERROR) — the fold-reset fix holds.
- **Magnitude collapse**: the action space has 3 magnitude bins (Quarter=0.25×, Half=0.50×, Full=1.00×). Observed usage across training: Quarter ≈ 99%, Half ≈ 0.2%, Full ≈ 0.5%. The model is mechanically biased to Quarter — it cannot size positions properly.
- **Sample-size floor**: 19–25 trades per val window is too few to compute a Sharpe with usable confidence intervals. 95% CI on Sharpe from 20 trades spans roughly ±15 annualized.
- **Training sharpe_ema oscillation**: epoch-to-epoch val Sharpe swings −18 → +6 → −15 etc. Anti-LR controller absorbs this (asymmetric fix from today) but the underlying oscillation means the policy itself is unstable.

### 1.2 What this spec does not address

This spec is sub-project A of a four-part production roadmap:

- **A — policy quality (this spec)** — does the model produce a real edge?
- B — validation rigor — can we prove the edge statistically? *(separate spec)*
- C — paper trading — does the edge survive realistic execution? *(separate spec)*
- D — production ops — can we deploy safely? *(separate spec)*

Paper trading integration, canary deployment, hyperopt re-runs, and architecture changes beyond the action space are **out of scope**.

---

## 2. Success criteria

The spec closes when a single L40S validation run (6-fold × 50-epoch, the existing walk-forward configuration) passes all **mandatory** gates and the final main commit is tagged `policy-quality-v1`.

### 2.1 Mandatory gates (all must pass to merge)

- **B-1 Multi-fold Sharpe:** Best val Sharpe (annualized) `> 10` on at least **5/6 folds**
- **B-2 Multi-fold WinRate:** Val WinRate `> 55%` on at least **5/6 folds**
- **SN Surrogate-noise:** Trained-model pooled Sharpe `>` 95th-percentile of N=30 random-action surrogate runs (`surrogate_noise_check.rs`). Uses pooled-across-6-folds val returns (~600 trades) so the comparison has statistical power.

### 2.2 Soft gates (ship with documented follow-up if missed)

- **A-1 Action-dist:** `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds (*substitute for H9-confirmed case in §7.2*)
- **A-2 WinRate range:** Val WinRate `≥ 55%` across folds
- **A-3 Trade count:** `≥ 100` trades per val window on **≥5/6 folds** (per-fold, not averaged)
- **CA Controller-activity:** No single controller fires in >50% of epochs — soft; load-bearing controllers go to sub-project B tech debt

### 2.3 Outcome paths

- **All mandatory + all soft pass** → tag `policy-quality-v1`, close spec clean.
- **All mandatory pass, some soft miss** → tag `policy-quality-v1`, open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` documenting deltas. Does not block sub-project B.
- **Any mandatory miss** → iterate on main (up to 2 more iterations, total 3 Phase-3 validation runs). Use `policy-quality-baseline` tag to revert if an iteration makes things worse. If all 3 iterations still miss a mandatory gate, the spec is **paused, not closed**. A dedicated triage spec is opened at `docs/superpowers/specs/YYYY-MM-DD-policy-quality-triage.md` with:
  - Summary of the 3 failed runs' evidence (which gates, by how much, which hypotheses correlate with failure)
  - Explicit diagnosis: policy/reward/state bottleneck? architecture bottleneck? data bottleneck?
  - Decision: continue on main with new hypotheses, OR scope a different angle (reward redesign, architecture change, alternative algorithm PPO/SAC) — the triage spec is a full brainstorming cycle that restarts the skill flow.

---

## 3. Architecture

Main-branch only. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that converge into the Phase-2 fix commit set. Tagged baseline for rollback.

```
main branch:

  [Phase 0]  commit(s): measurement substrate. Final commit tagged
      ↓        policy-quality-baseline. Baseline metrics captured here.

  [Phase 1]  Investigation. Temporary experiments in local worktree /
             git-stash — not committed (experiments are throwaway). Final
             output is triage documents committed to main.
      ↓

  [Phase 2]  commit(s): convergence fixes. Can be one commit or several
             per confirmed finding — main is our working branch. Total
             ~500 LOC expected; commit granularity is author's choice.
      ↓

  [Phase 3]  commit(s): validation iteration tweaks from 6-fold L40S run.
             Success-criteria gate checked.
      ↓

  [CLOSE]    Tag policy-quality-v1 on the final validated commit.
             If validation fails after 3 iterations, open triage spec
             per §2.3.
```

**Key architectural decisions:**

- **Main-branch workflow.** All commits land on main. ~500 LOC total is small enough that feature-branch isolation isn't needed.
- **Measurement first.** No fix lands until its detection is codified in a smoke test.
- **Phase 1 investigation stays local.** Experimental instrumentation lives in the worktree during measurement runs, never committed. Only the resulting triage documents commit to main.
- **Fixes can be multiple commits.** The spec's earlier "single convergence commit" rule is relaxed — each confirmed finding can be its own commit if that's easier to review. The discipline that remains: all fixes land before Phase 3 validation runs; each commit leaves smokes green.
- **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production.
- **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence.
- **Tagged baseline for rollback.** `git tag policy-quality-baseline` after Phase 0 completes. If Phase 2+ breaks main, `git reset --hard policy-quality-baseline` restores known state. Tag `policy-quality-v1` marks the validated close.

**Budget estimate:** ~4–6 days elapsed.

- L40S validation runs: ~1 hour each, budget 3 runs = **3 hrs** (one after Phase 2, up to 2 more to iterate if partial)
- Local smoke runs (RTX 3050): ~3 min each, budget unlimited
- Investigation runs during Phase 1 — 4 tracks × 2-3 dedicated measurement runs each:
  - Track 1 (magnitude): forced-exploration epoch experiment (~1 hr run) + baseline (~12 min) + per-hypothesis follow-up = ~1.5 hr
  - Track 2 (reward audit): 1 instrumented 20-epoch run (~12 min) is enough to measure all 8 terms
  - Track 3 (controllers): 1 instrumented run + 1 comparison run with selected controllers disabled = ~25 min
  - Track 4 (exploration): 1 instrumented run (~12 min)
  - Total investigation GPU: **~2.5 hrs**

Total GPU: **~5.5–6.5 hrs L40S**.

**Target GPU pool:** `ci-training-l40s` throughout. H100 only if L40S queue saturates (no expected architecture-specific differences).

---

## 4. Phase 0 — Measurement substrate

One or more commits on `main`. Adds smoke tests + HEALTH_DIAG extensions. **No behavioral change** — the new tests should PASS on current baseline (action-dist tests use relaxed smoke thresholds, not production thresholds).

After Phase 0 completes, tag the final Phase-0 commit `policy-quality-baseline` for rollback anchor.

### 4.1 New smoke tests (ignored, run manually)

All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-gated and runs via `FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- <name> --ignored --nocapture`.

**`magnitude_distribution.rs`** — Track 1 gate
- Train 20 epochs on fxcache smoke data.
- Assert: `F_Half ≥ 5%` AND `F_Full ≥ 5%` (relaxed from 15% success-criterion bar — this is a smoke gate, not a production gate).
- HEALTH_DIAG readback: per-magnitude Q means, var_scale mean, kelly_f mean, per-branch grad ratio, action distribution per bin.

**`reward_component_audit.rs`** — Track 2 gate
- Train 20 epochs.
- Reward terms fall into three measurement classes, each with its own detection:
  - **Additive** (R1 step_return, R5 micro-reward, R6 loss-aversion, R7 segment-patience): record `|term| / |total reward|` per step, assert the active term's median ratio ≥ 1% across epochs.
  - **Transform** (R2 PopArt): record `|post_popart − pre_popart|`; if < 10% of |pre_popart| typical scale, PopArt is no-op-equivalent and DELETE-triage.
  - **Sample-selector** (R3 CF-flip, R4 trailing-stop): record activation rate (fraction of samples/episodes the term fires on) AND downstream effect (grad-norm delta on fired vs unfired batches). Sub-noise = <1% firing rate OR indistinguishable grad signal.
- Terms below their class's threshold are triaged for DELETE in Phase 2.
- Implementation detail: the test uses one training run with instrumentation, not 8 separate runs.

**`controller_activity.rs`** — Track 3 gate
- Train 20 epochs, record per-controller firing rate.
- Assert: no single controller (anti-LR, adaptive-tau, adaptive-gamma, grad-clip, CQL-alpha, cost-anneal) fires in > 50% of epochs. High firing rate = load-bearing → production risk.

**`exploration_coverage.rs`** — Track 4 gate
- Train 20 epochs, record per-epoch magnitude-branch action entropy.
- Assert: entropy ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20. Collapse-speed check — if magnitude entropy hits 0 before epoch 20, something is quenching exploration.

**`multi_fold_convergence.rs`** — fast local variant of the full gate
- 3 folds × 20 epochs (vs 6 × 50 on L40S). Runs ~5 min on RTX 3050.
- Assert: all 3 folds complete without NaN, Best Sharpe > 0 on at least 2/3.

**`surrogate_noise_check.rs`** — false-positive detector
- Use the trained model from Phase 2 or Phase 3.
- **Pool all 6 validation folds** into a single val-return stream (~600 trades aggregated) — per-fold Sharpe is sample-size-noisy, so the per-fold comparison could be satisfied by luck. Pooled Sharpe has the statistical power needed for this gate.
- Generate **N=30 surrogate runs**: for each, REPLACE model actions with uniformly random actions (matched marginal action distribution to the trained model's, so the comparison isn't trivially won by action-frequency differences alone). Run backtest per surrogate, collect 30 random-action pooled Sharpes.
- Compute the 95th percentile of the surrogate Sharpe distribution.
- Assert: **trained-model pooled Sharpe > surrogate 95th-percentile Sharpe** (i.e., the trained model is outside the top 5% of what random actions produce by chance).
- Rationale for statistical gate: "0.5× ratio" in a single comparison is vulnerable to sample noise; a percentile-based test gives genuine statistical power with explicit false-positive control.
- Critical production-readiness gate. Permanent artifact — guards against future measurement regressions.

### 4.2 HEALTH_DIAG extensions

Add to the existing HEALTH_DIAG log line in `training_loop.rs`. All forward-only, no backward gradient, cost is one scalar readback per epoch per field.

Track 1 fields (magnitude diagnosis):

Covering H1–H5 (common causes):
- `q_mag_full`, `q_mag_half`, `q_mag_quarter` — mean Q-value per magnitude bin
- `var_scale_mean` — average Var[Q]-derived conviction shrink (H2)
- `kelly_f_mean`, `avg_win_ratio` — Kelly fraction + `avg_win / avg_loss` ratio (H3)
- `grad_ratio_mag_dir` — magnitude-head grad norm ÷ direction-head grad norm (H4)
- `action_dist_quarter`, `action_dist_half`, `action_dist_full` — epoch-level usage fractions (H5 indirect)

Covering H6–H10 (codebase-specific):
- `trail_fire_rate_{quarter, half, full}` — trailing-stop firing rate per magnitude bin (H6)
- `hold_time_at_exit_{quarter, half, full}` — mean bars held until exit, per magnitude (H6)
- `vsn_mask_{magnitude, direction}` — VSN mask mean per branch (H7)
- `noisy_sigma_{mag_head, dir_head}` — NoisyNets σ mean per branch's weights (H7)
- `target_drift_{mag_head, dir_head}` — cumulative ‖target − online‖ per branch (H8)
- `action_dist_eval_{quarter, half, full}` — magnitude usage during evaluation rollouts (distinct from training-mode `action_dist_*`) (H10)

Track 2 fields (reward V7 audit):
- `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return (see class-specific measurement in §4.1)

Track 3 fields (controllers V7 audit):
- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch
- `controllers_firing_fraction` — aggregate: fraction of epochs where ANY controller fired non-trivially

Track 4 fields (exploration V7 audit):
- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch
- `explore_entropy_direction` — same for direction branch (control variable)
- `noisy_net_sigma_mean` — average NoisyNets σ across weights

### 4.3 Phase 0 exit criteria

- All new smoke tests compile and produce output. The Phase 0 smokes use **relaxed thresholds** (e.g. magnitude ≥5% not 15%) so they pass current code; they will tighten in later commits or their production equivalent lives in the Phase 3 validation gate.
- HEALTH_DIAG fields appear in training logs with non-zero values across a baseline 20-epoch run.
- Baseline reference metrics recorded in `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md` and committed to main — audit trail.
- `git tag policy-quality-baseline` on the final Phase-0 commit + `git push origin policy-quality-baseline`.

---

## 5. Phase 1 — Parallel investigation tracks

**Branch discipline during Phase 1:**

- No **code** commits land on `main` during Phase 1 — the temporary instrumentation (e.g., forced-exploration epsilon=1.0 experiment, controller-disable experiments) is experimental and would pollute history.
- **Triage documents DO commit to `main`** as they're completed. Each track's triage doc is a permanent artifact.
- Experimental code changes live in the worktree (uncommitted) or in local `git stash` during a measurement run; discard after the triage doc is written.

This is the pragma: instrumentation commits from Phase 0 are kept (useful), investigation code is throwaway, triage output is committed.

### 5.1 Track 1 — Magnitude collapse diagnosis

Ten candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** verdict based on the measurement signal. H1–H5 are common RL-trading causes; H6–H10 are codebase-specific possibilities surfaced in spec review.

| # | Hypothesis | Detection signal | Detection threshold |
|---|---|---|---|
| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | Forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | `mean(R_full)/std(R_full)` is ≤ 0.5× `mean(R_quarter)/std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. |
| H2 | Var[Q] conviction-sizer over-shrinking — `effective_max_pos *= 1/(1 + sqrt(Var[Q]))` permanently collapses the target toward Quarter. | `var_scale_mean` field | `var_scale_mean < 0.3` across ≥10/20 epochs |
| H3 | **Kelly as hard multiplier is too aggressive for marginal edges** — Kelly formula `f = (bp − q)/b` gives small `f` when win rate × avg_win ≈ avg_loss (marginal edges, typical in trading). The code multiplies `effective_max_pos *= kelly_f`, shrinking the envelope before magnitude selection even sees it. Not a Kelly bug; a design question: should Kelly be a hard multiplier or a soft gate? | `kelly_f_mean` AND `avg_win / avg_loss` stats from training | `kelly_f_mean ≤ 0.25` AND `avg_win / avg_loss < 1.3` — confirms marginal-edge scenario where Kelly's low value is correct math but wrong policy tool. |
| H4 | Magnitude-head gradient starvation — the per-action-branch gradient (*not* the per-loss-component IQN/CQL/C51/Ens split) to the magnitude head is much smaller than to the direction head. Magnitude weights stay near init. | `grad_ratio_mag_dir` field, measured as end-of-epoch average of per-step ‖∇ W[mag_head]‖ ÷ ‖∇ W[direction_head]‖ (pre-clip). | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs |
| H5 | Bootstrap bias — initial random exploration anchored Quarter, epsilon decayed before the policy escaped. | Action entropy vs epsilon schedule | entropy < 0.3 before epoch 5 AND epsilon below 0.1 |
| H6 | Regime-adaptive trailing stop (wired 2026-04-21) closes Full-position trades prematurely. Full moves hit profit-retreat thresholds faster than Quarter moves because P&L magnitude is larger. | Per-magnitude trailing-stop firing rate, plus mean hold-time-at-exit per magnitude. | Trail firing rate on Full positions > 3× rate on Quarter positions, AND mean hold-time-at-exit on Full < 0.5× Quarter. |
| H7 | VSN (variable selection network) or NoisyNets-injected noise disproportionately masks/perturbs the magnitude-head's input features, so effective signal-to-noise is worse than direction head. | VSN mask distribution per branch (forward hook), NoisyNets σ per weight tensor. | Magnitude head mean-mask weight < 0.5× direction head, OR NoisyNets σ on w_b1fc > 2× other branch weights. |
| H8 | Target-network tau too slow for magnitude propagation — because magnitude distribution is concentrated at Quarter, target updates rarely propagate signals about Half/Full (never sampled enough), effective learning rate is lower for those bins. | Target-net weight drift per branch (tau × online_grad accumulated). | Magnitude-head target drift < 0.1× direction-head drift cumulative. |
| H9 | **Not a bug: data favors Quarter.** Most bars are low-volatility range; optimal sizing per bar IS Quarter. Removing magnitude branch entirely would be the correct design, not a fix. | Per-magnitude Sharpe during forced-exploration epoch (H1 measurement reused). | If all three magnitudes show similar Sharpe (no statistically significant difference), Quarter-collapse is correct behaviour and magnitude branch is dead code. |
| H10 | Entropy regularization + argmax tie-breaking: entropy bonus encourages uniform distribution across magnitudes; argmax at evaluation picks the first bin (Quarter by index) on any near-tie. Training-time exploration looks uniform but evaluation is quarter-biased. | Compare training-mode action distribution vs eval-mode action distribution. | Training-mode magnitude entropy > 0.8 (near-uniform) AND eval-mode `F_Quarter` > 90% (collapse at eval only). |

**Fix sketch (if confirmed):**

- H1 → rebalance reward gradient per magnitude (advantage weighting by bin, or per-bin loss scale)
- H2 → clamp `var_scale ≥ 0.5` OR remove entirely if redundant with Kelly
- H3 → replace Kelly hard multiplier with a soft gate (e.g., `if kelly_f < 0.3 then magnitude ≤ Half`, keeping magnitude head free to choose); OR remove Kelly entirely and accept greater variance in exchange for proper action space exploration
- H4 → rebalance 4-branch advantage weights OR adjust initial weight scale for the magnitude head
- H5 → extend exploration warmup for magnitude branch only
- H6 → feed per-magnitude scale into the trailing-stop distance (larger positions get proportionally wider stops), OR disable trailing stop entirely if H6 + H9 both confirmed
- H7 → mask VSN's effect on magnitude features; reduce NoisyNets σ on magnitude weights
- H8 → increase tau OR use per-branch target update rates (direction faster, magnitude slower with sample-weighted updates)
- H9 → delete magnitude branch, reduce action space from 108 to 36 (direction × order × urgency only). Massive simplification if this hypothesis confirms.
- H10 → break argmax ties by Q-value margin (no bin ordering bias); OR match training/eval distributions via stochastic eval rollouts

### 5.2 Track 2 — Reward V7 audit

Every reward signal gets the V7 3-step treatment.

**Reward terms currently active (verified at Phase 0):**

| # | Term | Current role |
|---|---|---|
| R1 | `step_return` — pure per-bar P&L | Base reward, always active |
| R2 | PopArt normalization | Scales `step_return` to unit-variance |
| R3 | Counterfactual flip (50% episodes) | Flip direction + negate features for symmetry |
| R4 | Trailing stop (regime-adaptive, wired today) | Forces exit on profit retreat |
| R5 | Micro-reward (price-confirm + book-agg + hold-quality) | Dense per-bar directional signal |
| R6 | Loss-aversion asymmetric clamp | Tanh-squash negative returns |
| R7 | Segment return patience multiplier | √(hold_time / expected_hold) |
| R8 | `raw_returns_out` vs shaped returns | Pure P&L vs shaped distinction |

**V7 audit template per term:**

1. **Identify signal** — what phenomenon does this target? Why was it added? (`git log -S` + existing specs)
2. **Check better-form coverage** — is there an existing mechanism that handles this invariant?
3. **Measure empirically** — inspect the term's contribution via `reward_contrib_<term>` over 20 epochs
4. **Decide** — sub-noise OR better-form covers → DELETE. Real signal, no coverage → KEEP (or WIRE PROPERLY if half-wired).

**Decision matrix per term:**

| Evidence | Decision |
|---|---|
| `reward_contrib_<term>` < 1% in majority of epochs | DELETE |
| Contribution is meaningful AND another term already covers the same invariant | DELETE with breadcrumb |
| Contribution is meaningful AND invariant is not otherwise covered | KEEP |
| Contribution appears meaningful but code is half-wired (e.g., forward only, no backward) | WIRE PROPERLY or DELETE based on backward-completion effort |

### 5.3 Track 3 — Controllers V7 audit

Every adaptive controller currently shaping training gets a V7 3-step treatment.

**Controllers currently active:**

| # | Controller | Current role |
|---|---|---|
| C1 | Anti-intuitive LR (asymmetric, wired 2026-04-21) | Fires every epoch after warmup; boosts LR on good signal, dampens on sustained bad |
| C2 | Adaptive tau (target-net update rate) | Adjusts based on Q-drift |
| C3 | Adaptive gamma (discount) | Curriculum: low gamma early, high later |
| C4 | Adaptive grad clip | Threshold adjusts to running grad-norm distribution |
| C5 | CQL alpha schedule | Pessimism weight vs online data confidence |
| C6 | Cost-anneal schedule (`cost_anneal_ptr`) | Tx-cost curriculum: 0 early, full later |
| C7 | Learning-rate scheduler (base) | Step decay or cosine |

**Audit question per controller:** is this load-bearing (firing often, without it training breaks) or diagnostic (firing rarely, just catching edge cases)? A production-deployable policy should have controllers be diagnostic.

**Decision matrix per controller:**

| Evidence | Decision |
|---|---|
| Firing rate > 50% of epochs AND disabling it breaks training | LOAD-BEARING — keep, document the necessity, flag as production-readiness concern |
| Firing rate < 10% of epochs | DIAGNOSTIC — keep, no action |
| Firing rate 10–50% AND disabling it doesn't break training | CANDIDATE FOR DELETE — measure impact of removal, delete if within noise |
| Controller's trigger condition is symptom of upstream bug | ROOT-CAUSE FIX — fix upstream, controller becomes diagnostic |

**Track 3 smoke test deliverable:**
```rust
// crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs
fn test_controllers_not_load_bearing() {
    // Train 20 epochs. Record per-controller firing rate.
    // Assert: no single controller fires in > 50% of epochs.
    // High firing = policy can't hold on its own = production risk.
}
```

**HEALTH_DIAG extensions (Track 3):**
- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch, aggregated over window

### 5.4 Track 4 — Exploration V7 audit

**Exploration mechanisms currently active:**

| # | Mechanism | Current role |
|---|---|---|
| E1 | Epsilon-greedy | Random action with probability ε, decays over training |
| E2 | NoisyNets | Injects parametric noise in fc layers for exploration |
| E3 | Count-based bonus (`count_bonus=true`) | Reward bonus for rarely-visited (state, action) pairs |
| E4 | Entropy regularization | Pushes policy toward uniform action dist |
| E5 | Counterfactual flip (R3) | Data-level exploration via mirroring (already in Track 2) |

**Audit question per mechanism:** are these redundant with each other, and is the net exploration pressure at the action/magnitude level appropriate?

**V7 audit template per mechanism:**
1. Identify signal — what exploration problem does this solve?
2. Check better-form — does another mechanism already provide this?
3. Measure — log magnitude-level action entropy, "novel state" count, per-mechanism contribution estimate
4. Decide — DELETE redundant, TUNE under-powered, KEEP covering unique problem

**Track 4 smoke test deliverable:**
```rust
// crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs
fn test_exploration_covers_magnitude_space() {
    // Train 20 epochs. Record per-epoch magnitude-level action entropy.
    // Assert: entropy over magnitudes ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20.
    // If entropy collapses too fast, magnitude space is under-explored.
}
```

**HEALTH_DIAG extensions (Track 4):**
- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch
- `explore_entropy_direction` — same for direction branch (control)
- `noisy_net_sigma_mean` — average NoisyNets σ across weights

### 5.5 Phase 1 exit

Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item. Four track outputs consolidate into one Phase 2 commit.

**Conflict resolution when tracks disagree:**

Tracks can reach conflicting conclusions — e.g. Track 1-H6 might say "fix trailing stop for magnitude" while Track 2 measures trailing stop contribution as sub-noise and triages DELETE. Conflict-resolution order:

1. **Mandatory gate dominance.** A fix that removes an item blocks a mandatory-gate-relevant mechanism only if measurement shows the mechanism isn't contributing to the mandatory outcome. In the trailing-stop example: if Track 2 shows R4 is sub-noise, DELETE wins regardless of Track 1 wanting to fix it — "fix a dead thing" is pointless.
2. **Simplification wins.** Among equally-justified options, pick the one that reduces code surface (DELETE > FIX > ADD).
3. **Escalation.** If (1) and (2) don't resolve, the spec author decides and documents the call in the Phase 2 commit.

---

## 6. Phase 2 — Convergence fix commits

Commits on `main` — one or several, author's choice. All Phase 1 findings fixed before Phase 3 validation begins. Each commit leaves smokes green.

**Commit structure:**

```
fix(dqn): policy-quality V7 audit — <N> bugs, <M> dead terms removed

Fixes confirmed by Track 1 (magnitude collapse):
  - H<id> (<hypothesis>): <fix>
  - ...

Deleted by Track 2 (reward V7 audit):
  - R<id> (<term>): sub-noise / better-form covered, deleted

Kept by Track 2 (measurement confirmed real signal):
  - R<id>: retained, <evidence summary>

Fixes confirmed by Track 3 (controllers V7 audit):
  - C<id> (<controller>): <root-cause fix OR removed>
  - Note if any controller remains LOAD-BEARING — flagged for sub-project B follow-up.

Fixes confirmed by Track 4 (exploration V7 audit):
  - E<id> (<mechanism>): <tune OR delete>

Smoke tests added (Phase 0):
  - magnitude_distribution     (Track 1)
  - reward_component_audit     (Track 2)
  - controller_activity        (Track 3)
  - exploration_coverage       (Track 4)
  - multi_fold_convergence     (fast local variant of Phase 3 gate)
  - surrogate_noise_check      (false-positive detector)

Validation gate (Phase 3):
  - 6-fold × 50-epoch L40S run, success criteria A+B+surrogate-noise pending
```

**Rules:**

- **Commits land on main.** Author chooses granularity (one per finding, or grouped). Each commit leaves main in a working state.
- **All smoke tests green before push on every commit:** existing (`td_propagation`, `test_50_epoch_convergence`) + all six Phase 0 smokes (`magnitude_distribution`, `reward_component_audit`, `controller_activity`, `exploration_coverage`, `multi_fold_convergence`, `surrogate_noise_check`).
- **No config-file-only changes.** Every behavioral change has source code attached.
- **Commit messages reference the triage finding** (e.g., "fix(dqn): H2 — clamp var_scale ≥ 0.5 per track1 triage"). Makes git log readable as the audit trail.

---

## 7. Phase 3 — Validation

### 7.1 Validation run

```bash
./scripts/argo-train.sh dqn --gpu-pool ci-training-l40s --epochs 50 --baseline
```

Existing walk-forward config produces 6 folds on `test_data/futures-baseline`.

### 7.2 Pass/fail gate

| Gate | Pass condition | Source |
|---|---|---|
| A — Action-dist (soft) | `F_Half ≥ 15%` AND `F_Full ≥ 15%` on ≥4/6 folds | HEALTH_DIAG `action_dist_*`, absence of LOW EXPOSURE DIVERSITY warnings |
| A — WinRate (soft) | Val WinRate ≥ 55% across folds (upper-bound fine) | Metrics kernel |
| A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel |
| **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log |
| **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel |
| **Surrogate-noise (MANDATORY)** | Trained-model pooled Sharpe > 95th-percentile of N=30 surrogate random-action runs | `surrogate_noise_check.rs` smoke |
| Controller-activity (soft) | No controller fires in > 50% of epochs | HEALTH_DIAG `controller_fire_*` fields |

Criterion B gates AND surrogate-noise check MUST pass for the spec to close successfully and merge to main (Phase 4). Criterion A gates are soft — if B and surrogate-noise pass but some A gate misses, the spec closes with a documented A follow-up. Controller-activity is soft — high firing rates go into a tech-debt follow-up note (sub-project B concern).

**Special case for H9 confirmed (magnitude branch deleted):** if Phase 1 H9 confirms and Phase 2 removes the magnitude branch, Criterion A's `F_Half ≥ 15%` and `F_Full ≥ 15%` become vacuous (no bins to distribute across). Replace with substitute gate: trade count AND WinRate criteria still apply; the action-distribution gate becomes `direction bin distribution healthy` (F_Short + F_Long each ≥ 20% of non-Flat actions). The spirit of Criterion A — "model actually trades, doesn't collapse to a single action" — is preserved.

### 7.3 Outcome handling

See §2.3 for the outcome paths. Concretely:

- **Missed mandatory gate** → iterate up to 2 more validation runs with additional commits on main. If 3 runs total still miss a mandatory gate, escalate out of this spec (triage doc per §2.3).
- **Missed soft gate only** → tag `policy-quality-v1`; open `docs/superpowers/specs/YYYY-MM-DD-policy-quality-followup.md` with: which gate missed, by how much, what hypothesis was wrongly rejected (if applicable), proposed next-spec scope.

### 7.4 Closing the spec

When mandatory gates pass:

```bash
git tag -a policy-quality-v1 -m "Policy quality V7 audit — sub-project A closed"
git push origin policy-quality-v1
```

Update the top of this design doc with `Status: CLOSED`, reference the Phase-3 results doc, and mark sub-project A complete in any production-readiness tracking.

### 7.5 Permanent artifacts (survive spec closure)

Left in the tree after the spec closes:

- `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs`
- `crates/ml/src/trainers/dqn/smoke_tests/reward_component_audit.rs`
- `crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs`
- `crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs`
- `crates/ml/src/trainers/dqn/smoke_tests/multi_fold_convergence.rs`
- `crates/ml/src/trainers/dqn/smoke_tests/surrogate_noise_check.rs`
- HEALTH_DIAG extensions across all 4 tracks (all behavioral-code-path reusable)
- `git tag policy-quality-baseline` (Phase 0 anchor for rollback)

These are the regression coverage for the invariants this spec establishes.

---

## 8. Out of scope

- Paper trading integration → sub-project C
- Canary deployment, hot-swap, A/B testing → sub-project D
- Hyperopt re-runs → separate concern; current defaults are used as baseline
- Architectural changes beyond action space (no new network heads, no new layers)
- State representation changes (OFI features, regime features — kept as-is)
- Hyperopt-params file regeneration. If Phase 2 changes action-space layout (e.g., adds a 4th magnitude bin), the existing `hyperopt_params.json` may have stale branch sizes. Policy: detect mismatch at runtime, log warning, use CLI branch-size defaults. A proper hyperopt re-run is a separate concern (sub-project B includes it implicitly as part of multi-fold iteration).

---

## 9. Risk register

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Phase 1 finds all 10 Track-1 hypotheses REJECTED | Low | High | Track 2/3/4 almost certainly surface findings; if all 4 tracks come up empty, magnitude collapse might be **H9 (not a bug, data favors Quarter)**. Delete magnitude branch per H9 fix sketch — action-space simplification is still a production win. |
| H9 confirmed: magnitude branch deletion breaks backwards compat more than expected | Medium | Medium | Greenfield policy covers this — we accepted retrain cost. Smoke tests validate pre/post equivalence in non-magnitude dimensions. |
| Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. |
| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Iterate on main with additional fix commits. Baseline tag `policy-quality-baseline` is rollback anchor. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). |
| Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. |
| Iteration cost exceeds ~7 hrs L40S (budgeted 5.5–6.5 for plan + 1 buffer) | Low | Low | Iteration budget hard-capped at 3 Phase-3 validation runs (3 hrs). If the 3rd still misses a mandatory gate, the spec escalates — no further validation runs on this branch. |
| Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. |
| Controller-audit (Track 3) finds multiple load-bearing controllers that can't be removed in this spec | Medium | Medium | Document as tech-debt, flag for sub-project B. Spec can still close if policy is profitable. But "load-bearing controllers = not production-deployable" remains a known concern. |

---

## 10. References

- Prior specs: `2026-03-23-profitability-roadmap.md` (P5–P11, some superseded)
- V7 methodology memory: `feedback_v7_gem_methodology.md`
- Stability fixes that preceded this spec:
  - commit `932ac2bda` — graph-capture dtoh fix
  - commit `679881fa5` — v_range config clamp
  - commit `a1346dac1` — fold-boundary EMA reset
  - commit `f1359f3dc` / `408583145` — asymmetric anti-LR
  - commit `8afe9562e` — sccache in ensure-binary
