# SP21 — Train/Eval Coherence + ISV Defrost

**Author**: jgrusewski + assistant pair audit
**Date**: 2026-05-10
**Branch**: `feat/sp18-combined` (off `sp19-20-wr-first` worktree)
**Predecessors**: SP19/SP20 (Phase 5 aux→Q gate, 4-quadrant reward, imbalance bars wiring)

## Meta-finding

A pair audit of MinIO-archived training logs (`xmd6b` 2026-05-08, 30 epochs; `d7bj7` 2026-05-10, 2 epochs) revealed three intertwined failure modes that explain why every SP from SP3-SP20 has produced WR pinned at 50% and PF≈1.00 in the surfaced training metrics — **but the policies have actually had asymmetric edge all along**.

### What was hidden

Across 30 epochs of `xmd6b`:

```
Epoch 0: val [sharpe=81.91 win_rate=0.4594 profit_factor=1.328 total_pnl=0.559 trades=139251 active_frac=0.560]
Epoch 5: val [sharpe=59.76 win_rate=0.4645 profit_factor=1.232 total_pnl=0.248 trades=100118 active_frac=0.301]
Epoch 30: val [sharpe=25.54 win_rate=0.4814 profit_factor≈1.18 total_pnl≈0.16  trades≈55000  active_frac≈0.10]
```

The policy is `WR=46-48%, PF=1.18-1.33` — **asymmetric-payoff strategy** (more frequent small losers, fewer larger winners). The previously-tracked `WR=50.0%, PF=1.00` came from training-rollout metrics with Thompson exploration noise + cost curriculum, which average to break-even regardless of what the learned policy is doing.

### What's broken

Three intertwined failure modes:

1. **Wrong-metric early stopping**: patience-based stopping watches `epoch_sharpe` (training-rollout, exploration-noisy) instead of `val_loss` (deterministic backtest). Training ran 30 epochs past peak val (epoch 2: Sharpe=90, total_pnl=0.44), throwing away ~70% of the alpha by training-induced overfitting.

2. **Frozen ISV controllers**: 6+ ISV slots stuck at sentinel/floor values across all epochs (`wr_ema=0`, `hold_pct_ema=0`, `hold_reward_ema=0`, `loss_cap=-1` sentinel, `hold_cost_scale` and `aux_conf_threshold` at floor). Per `feedback_isv_for_adaptive_bounds.md`, these controllers should be driving adaptive behavior — they're not adapting at all.

3. **Fake-trade fabrication**: enrichment pipeline (`extract_eval_trades_from_metrics`) FABRICATES per-trade data from aggregate stats with hardcoded placeholders (`60000.0, 0.0, 0.5`) at the call site. Whatever "self-improving enrichment" feeds back into hyperparams is driven by inputs that don't reflect reality.

### Goal of this superproject

Restore train/val metric coherence and defrost ISV controllers so the system can actually converge on its intended adaptive equilibrium. This is foundational, not optimization — without these fixes, every previous SP's "improvements" were evaluated against blind metrics.

### Canonical signal-driven pattern (template for Tier 3 fixes)

The curiosity-pressure ISV slot (`ISV[CURIOSITY_PRESSURE_INDEX=346]`, SP11 Fix 39) is the **reference implementation** for what every signal-driven slot should look like. When fixing Tier 3 ISVs, mirror this pattern:

1. **Producer kernel** computes from a real upstream signal — not a constant. Curiosity reads val-sharpe trend + magnitude-ratio dispersion.
2. **Pearl-A first-observation bootstrap** — sentinel value (0.0 or designated sentinel) replaced directly on first valid observation, NOT blended (per `pearl_first_observation_bootstrap.md`).
3. **Wiener-α EMA** for steady-state updates (per `pearl_wiener_optimal_adaptive_alpha.md`), or fixed slow blend if no variance EMA available.
4. **Fractional upper bound** scaled to a related magnitude EMA — curiosity uses `0.3 × pnl_reward_magnitude_ema`. Prevents runaway controllers.
5. **Permanent floor** as a fraction of bound — curiosity uses `0.2 × bound` (per `pearl_blend_formulas_must_have_permanent_floor.md`). Prevents deadlock on signal disappearance.
6. **HEALTH_DIAG visibility** — slot value printed every epoch so freezes are observable. Curiosity is in the `sp11_reward [...curiosity_p=...]` line.

**Verification that the curiosity pattern is alive** (xmd6b logs):
```
HEALTH_DIAG[0]: ... curiosity_p=0.000 ...    ← Pearl-A sentinel bootstrap
HEALTH_DIAG[1]: ... curiosity_p=0.109 ...    ← live, post-bootstrap
HEALTH_DIAG[2]: ... curiosity_p=0.406 ...    ← rising as trend signal develops
```

If a Tier 3 fix doesn't show this same epoch-1-bootstrap-then-live trajectory, the fix is incomplete.

### Cross-cutting principle

**Every threshold or bound in training control flow must be signal-driven from an ISV slot, not hardcoded.** Hardcoded constants found during this audit:

| Site | Constant | Should derive from |
|---|---|---|
| `early_stopping.rs:59` `min_delta` | `0.001` | `ISV[VAL_SHARPE_VAR_EMA_INDEX=351]` (T1.1b) |
| `enrichment.rs:17-26` epsilon adapter | `2.0`, `0.5`, `-0.5`, `0.8`, `0.95`, `1.2` | val_sharpe distribution percentiles (T1.3) |
| `enrichment.rs:40` gamma blend | `5.0`, `[0.85, 0.98]` | hold-time distribution (T1.3) |
| `enrichment.rs:57` magnitude filter | `pnl < -0.001` | per-trade pnl variance EMA (T1.3) |
| `enrichment.rs:61` holding-bars filter | `< 3` | observed avg hold time ISV slot (T1.3) |
| `enrichment.rs:67` direction-rate clamp | `[0.5, 2.0]` | per-direction WR variance (T1.3) |
| `enrichment.rs:81` ensemble growth | `× 1.1` | ensemble agreement variance (T1.3) |
| `mod.rs:142` `min_improvement_rate` | `0.1` | val_sharpe variance EMA (T1.4) |
| `training_loop.rs:7425` save threshold | `improvement_rate > 0.01` | val_sharpe variance EMA (T1.4) |
| `training_loop.rs:2187` regime threshold | `adx_threshold = 30.0` | ADX distribution EMA (Tier 5) |
| `training_loop.rs:987` cost-anneal schedule | `1/(1+exp(-((epoch-10)/3)))` — `10`, `3` | training-stability EMA (T1.5) |
| `trade_physics.cuh:714` capital floor | `max_equity × 0.75f` (25% DD) | drawdown distribution EMA (T1.6) |
| `state_layout.cuh:317` `MIN_HOLD_TARGET` | `30.0f` zombie #define (claimed replaced by ISV slot 460 per Class A audit, but still exists) | DELETE — `feedback_no_legacy_aliases` (T2.4) |
| `config.rs:1479-1480` PER schedule | `per_alpha=0.6, per_beta_start=0.6` | priority distribution variance EMA (T2.5) |
| `experience_kernels.cu:450-454` cost curriculum ranges | `PHILOX_UNIFORM(0.5, 2.5)`, etc. | broker-physics priors — fundamental constants, NOT tuning (no fix needed; but document the rationale) |
| `c51_loss_kernel.cu:13,28` | `MAX_PER_SAMPLE_CE=50.0`, `LABEL_SMOOTHING_BASE=0.02` | label-smoothing already has `pearl_loss_balance_controller` infrastructure; CE clamp from gradient norm distribution |
| ~~`gpu_curiosity_trainer.rs:342,401`~~ | ~~`let beta: f32 = 0.0`~~ | **REMOVED 2026-05-10**: false alarm. These are cuBLAS `cublasSgemm_v2` β parameters (`C = α·A·B + β·C` with β=0 = "don't accumulate"), unrelated to curiosity reward weight. Actual curiosity is `ISV[CURIOSITY_PRESSURE_INDEX=346]`, already ISV-driven per SP11 Fix 39, and confirmed firing in xmd6b logs (`curiosity_p: 0.000 → 0.109 → 0.406` across epochs 0-2). |

This principle is the SP21 unifying invariant. Per `feedback_adaptive_not_tuned.md`: *"fixes are signal-driven (EMA/windowed), not tuned constants."* If we accept that principle, ALL the above must be ISV-driven; if we reject it for any specific case, the rejection must cite a concrete reason (e.g., fundamental floor per `feedback_isv_for_adaptive_bounds.md`).

---

## Findings catalog

### Tier 1 — Wrong-source + signal-driven fixes

| # | Finding | Site | Root cause | Fix |
|---|---|---|---|---|
| T1.1a | Patience early-stopping reads `epoch_sharpe` (training rollout) instead of `val_loss` | `crates/ml/src/trainers/dqn/trainer/training_loop.rs:7234` | Copy-paste bug — the comment at `training_loop.rs:1499-1500` explicitly says "Use val_Sharpe (deterministic backtest), NOT epoch_sharpe". Backtracking honors this; patience early-stop doesn't. | Change `-log_output.epoch_sharpe` → `log_output.val_loss`. The latter is already `-val_sharpe` per line 1501; passing directly works for "lower is better" convention. |
| T1.1b | `EarlyStopping::min_delta` is a hardcoded constant `0.001` — violates `feedback_adaptive_not_tuned` | `crates/ml/src/trainers/dqn/early_stopping.rs:31-67` (struct), `crates/ml/src/trainers/dqn/trainer/constructor.rs:684` (construction site) | min_delta as constant means a noisy val_loss with σ=0.5 looks "improving" with min_delta=0.001 and we never trigger; a stable val_loss with σ=0.0001 would trigger spuriously. The threshold must scale with the noise floor. | Refactor `should_stop(val_loss, epoch)` → `should_stop(val_loss, min_delta, epoch)`. Compute `min_delta` per-call from `ISV[VAL_SHARPE_VAR_EMA_INDEX=351]` (variance EMA already produced by `launch_sp11_val_sharpe_delta_compute` per `training_loop.rs:6421`). Formula: `min_delta = k × sqrt(ISV[VAL_SHARPE_VAR_EMA])` where `k≈1.0` (1σ improvement counts as real). Bootstrap: when ISV slot is at sentinel/zero, fall back to a permanent floor per `pearl_blend_formulas_must_have_permanent_floor`. |
| T1.2 | Enrichment caller passes hardcoded `(60000.0, 0.0, 0.5)` placeholders for trade_count, total_pnl, win_rate | `training_loop.rs:1514-1521` | TODO never finished — actual values exist in val `WindowMetrics` but caller used constants | Wire `actual_val_metrics.trade_count, actual_val_metrics.total_pnl, actual_val_metrics.win_rate` through. These are observations, not bounds — no adaptation needed, just stop fabricating them. Verify the val pipeline exposes these on the consumed-loss path (we saw them in HEALTH_DIAG output, so they're being computed and just need to be plumbed to the enrichment call site). |
| T1.3 | Enrichment functions (E1-E8) contain ~10 hardcoded thresholds + magic-number multipliers for epsilon adaptation, gamma blending, magnitude filters, ensemble agreement | `enrichment.rs:17-26, 40, 57, 61, 67, 81, 156, 230` (entire 381-line file) | Per `feedback_adaptive_not_tuned`, every gating threshold should be signal-driven. The enrichment file was written quickly and never adapted. | **DEPENDS on T2.2 outcome** — if we delete `extract_eval_trades_from_metrics`, the enrichment function rewrites must concurrently signal-drive each threshold from the appropriate ISV slot OR a per-call computed quantile/variance. Specifics per threshold listed in cross-cutting principle table above. |
| T1.4 | Backtracking improvement-rate thresholds hardcoded | `training_loop.rs:7425` (`> 0.01`), `trainer/mod.rs:142` (`min_improvement_rate: 0.1`) | Single-fold backtracking save criteria. Not signal-driven. | Both should derive from `ISV[VAL_SHARPE_VAR_EMA_INDEX=351]`. Save threshold = `k1 × sqrt(var_ema)`, route-acceptance threshold = `k2 × sqrt(var_ema)` with k2 > k1 (route acceptance is stricter than save trigger). |
| T1.5 | Cost-anneal schedule hardcoded sigmoid | `training_loop.rs:987` — `1/(1+exp(-((epoch-10)/3)))` | The `10` (transition epoch) and `3` (slope) gate the entire cost curriculum schedule. Choosing them by tuning rather than signal means we don't know if the policy is ready for full cost when full cost arrives. | Drive transition from training-stability EMA — when val_Sharpe variance is below threshold for N consecutive epochs, ramp cost. Concretely: read `ISV[VAL_SHARPE_VAR_EMA]`, compute `cost_anneal = sigmoid((stability_score - 0.5) × k)` where `stability_score = exp(-var_ema/var0)` for some signal-derived `var0`. |
| T1.6 | Capital floor hardcoded at 75% of peak equity | `trade_physics.cuh:714` (`max_equity * 0.75f`) | 25% drawdown trigger is a fundamental risk-management threshold AND a learning-signal gate (capital_floor breach forces episode boundary). Domain-specific but configurable. | This is a judgment call: is 25% DD a *fundamental* floor (regulatory/broker reality) or a *tuning* parameter? Recommendation: keep `0.75f` as fundamental (broker margin call territory) but add a comment explaining it. NO fix unless we find evidence it's incorrect for the actual broker integration. |

**Success criterion**: training run shows `🛑 Early stopping triggered` line near the val_Sharpe peak (within `patience` epochs), `min_delta` value at each call is logged (or asserted) to be derived from ISV slot 351 (not constant 0.001), and enrichment receives non-placeholder values (verifiable by adding an assertion or log line printing what enrichment received).

**Pearl compliance**:
- T1.1a follows `feedback_no_partial_refactor` (the comment-says-one-thing-code-does-another inconsistency must close)
- T1.1b follows `feedback_isv_for_adaptive_bounds` (threshold is an adaptive bound) and `feedback_adaptive_not_tuned` (signal-driven, not constant) and `pearl_blend_formulas_must_have_permanent_floor` (sentinel handling)
- T1.2 follows `feedback_no_todo_fixme` (placeholder constants in production code) and `feedback_no_stubs` (fabricated inputs are stub-shaped)

---

### Tier 2 — Architectural debt requiring design decisions

| # | Finding | Site | Decision needed |
|---|---|---|---|
| T2.1 | `check_early_stopping(avg_q_value)` — separate stop path on Q-value magnitude | `crates/ml/src/trainers/dqn/trainer/metrics.rs:436` | Q-value magnitude is not a learning signal — high Q can mean high alpha OR catastrophic value-explosion. Three options: (a) delete entirely (subsumed by val-loss path); (b) repurpose as Q-explosion guard with **signal-driven** threshold from `ISV[Q_VALUE_VAR_EMA]` per `feedback_isv_for_adaptive_bounds`; (c) keep but rename so its purpose is clear. Recommendation: **(a) delete** — `feedback_no_legacy_aliases` says no deprecated wrappers. If we keep it as a Q-explosion guard, it MUST be signal-driven (no hardcoded threshold). |
| T2.2 | `extract_eval_trades_from_metrics` fabricates per-trade tape from aggregate stats; downstream enrichment functions are themselves constants-soup (T1.3); 5 of 8 enrichment outputs (E1/E4/E6/E7/E8) computed-and-discarded since SP3+ | `crates/ml/src/trainers/dqn/trainer/enrichment.rs:99-138` (synth) + entire 381-line file | Function is mathematically a fiction — generates synthetic trades that "average to" the input aggregates. 3 of 8 outputs consumed (E2/E3/E5); 5 of 8 (E1/E4/E6/E7/E8) require per-trade data that never made it past the fake-trade synthesizer. | **DECISION 2026-05-10 (user-directed)**: option (a) — full per-trade tape emission + wire all 8 enrichments to real consumers. Multi-session scope (~5-6 days). See **T2.2 multi-phase scope** below. |
| T2.3 | `EarlyStopping::should_stop` test signature changes (T1.1b) | `crates/ml/src/trainers/dqn/early_stopping.rs:158, 183, 200, 216, 236, 250` (6 unit tests) | Refactoring `should_stop(val_loss, epoch)` → `should_stop(val_loss, min_delta, epoch)` breaks the test signatures. | Update all 6 tests to pass explicit `min_delta` per call. Per `feedback_no_partial_refactor`, this is part of the atomic T1.1b commit, not a separate task. |
| T2.4 | `MIN_HOLD_TARGET=30.0f` zombie #define after Class A audit | `state_layout.cuh:317` | Comment at lines 177, 296-322 explicitly says this constant was REPLACED by ISV slot 460 (`MIN_HOLD_TEMPERATURE_ADAPTIVE_INDEX`). But the #define still exists. Either: (a) zombie code that should be deleted; (b) used as fallback when ISV at sentinel — comment line 320 hints at fallback usage. Audit needed. | Trace usages of `MIN_HOLD_TARGET` macro. If only used via the fallback path, rename to `MIN_HOLD_TARGET_FALLBACK` and document. If unused, DELETE per `feedback_no_legacy_aliases`. |
| T2.5 | PER `per_alpha=0.6` and `per_beta_start=0.6` hardcoded | `config.rs:1479-1480` | PER (Prioritized Experience Replay) parameters control training-data sampling distribution. The original Schaul et al. paper uses these as fixed but other works adapt them. With `feedback_always_per` mandating PER ON, the *parameters* are still fixed. | Two options: (a) keep fixed as paper-canonical (but then the hyperparams should be derived from a research citation, not a magic number); (b) adapt from priority distribution variance EMA. Recommendation: **keep fixed for SP21**, but file the question for a separate SP if PER tuning is later identified as a leverage point. Document choice in `pearl_per_alpha_beta_paper_canonical`. |
| ~~T2.6~~ | ~~Curiosity trainer beta=0.0 hardcoded — silently disabled?~~ | ~~`gpu_curiosity_trainer.rs:342, 401`~~ | **WITHDRAWN 2026-05-10 (false alarm)**: the `beta=0.0` is the cuBLAS `cublasSgemm_v2` β API parameter (`C = α·A·B + β·C`), not a curiosity reward weight. Actual curiosity coefficient is `ISV[CURIOSITY_PRESSURE_INDEX=346]`, produced by SP11 Fix 39's `reward_subsystem_controller_kernel`, already signal-driven per `pearl_blend_formulas_must_have_permanent_floor`. Confirmed firing in xmd6b logs: `curiosity_p: 0.000 → 0.109 → 0.406` across epochs 0-2 (Pearl-A bootstrap from sentinel, then live). | **No fix needed.** System working as designed. |

**Success criterion** (T2.1): file no longer contains `check_early_stopping(avg_q_value, ...)`; tests that exercised it migrated or deleted. (T2.2): `extract_eval_trades_from_metrics` removed; enrichment functions consume real per-trade tape from `gpu_backtest_evaluator`; all 8 outputs wired to real consumers; all thresholds signal-driven from ISV.

#### T2.2 multi-phase scope (multi-session)

Phase decomposition for the per-trade-tape-driven enrichment refactor:

- **Phase 1**: Per-trade tape emission from `gpu_backtest_evaluator`. New buffers `[MAX_TRADES_PER_WINDOW]` × 7 fields (pnl, predicted_q at entry, direction, magnitude, holding_bars, ensemble_var at entry, bar_index). New CUDA kernel writes at trade-close events. Host readback path. Mapped-pinned for streaming. Sentinel/cold-start handling.
- **Phase 2**: Replace `extract_eval_trades_from_metrics` with real tape consumer. Drop the fabrication.
- **Phase 3**: Wire E1 (q_correction). New ISV slot `Q_BIAS_CORRECTION_INDEX`. Apply as additive Q-target offset at C51 loss + Bellman target.
- **Phase 4**: Wire E4 (per-branch LR scaling). Apply via existing per-group Adam LR infrastructure (`pearl_per_group_adam_hyperparams`).
- **Phase 5**: Wire E6 (winner indices) → PER priority bumps. Use existing PER infra to mark indices high-priority.
- **Phase 6**: Wire E7 (hindsight) → replay buffer synthetic experience injection.
- **Phase 7**: Wire E8 (curriculum weights) → segment sampling weights at episode start.
- **Phase 8**: Signal-drive ALL enrichment thresholds (replace hardcoded `2.0`, `0.5`, `1.5×`, etc. with ISV-driven from `val_sharpe_var_ema`, ensemble disagreement EMA, etc.).
- **Phase 9**: GPU oracle tests for each new code path.

Each phase is its own atomic commit (per `feedback_no_partial_refactor` — phase-internal contract migrations are atomic; cross-phase wiring is sequential). Phases 1-2 are mandatory together (can't have consumer without producer); Phases 3-7 can land independently in any order after Phase 2.

**Pearl compliance**: T2.1 honors `feedback_no_legacy_aliases`. T2.2 honors `feedback_no_stubs` (fake-trade synthesis is a stub-shaped pattern — produces shaped output without real signal).

---

### Tier 3 — Frozen ISV slots (defrost)

Each ISV slot below stayed constant across all 3 observed epochs of `d7bj7`. The producer kernel for each must be located, the freeze cause identified, and the fix applied per relevant pearl.

| # | ISV slot | Observed | Expected | Producer kernel (to locate) |
|---|---|---|---|---|
| T3.1 | `wr_ema` | 0.0000 (frozen) | EMA of segment-close win rate ∈ [0, 1] | `pearl_per_bar_vs_segment_pnl_signal_mismatch` patch added `is_win_per_env` buffer; verify producer is reading it AND writing to `wr_ema_idx` |
| T3.2 | `hold_pct_ema` | 0.0000 (frozen) | EMA of fraction of bars where policy holds an active position | Likely an EMA producer reading from action_counts or position_active_mask |
| T3.3 | `hold_reward_ema` | 0.0000 (frozen) | EMA of opportunity-cost reward magnitude per bar | Likely populated alongside hold_pct |
| ~~T3.4~~ | ~~`loss_cap`~~ | ~~-1.0000 (sentinel)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the value `-1.0` is NOT a sentinel — it's the LOSS_CAP controller's correct output for `WR_EMA < 0.50` (`-1.0 - clamp((wr_ema-0.50)/0.05, 0, 1)`). The actual sentinel per `state_reset_registry` is `0.0` (FoldReset bootstrap). After T3.1 fix, WR_EMA moves to actual val WR ≈ 0.46, but LOSS_CAP stays at -1.0 because WR is still below the 0.50 ramp threshold. **Will move to -2.0 only when WR_EMA crosses 0.55** (linear ramp in [0.50, 0.55]). GPU test `loss_cap_ramp_boundaries` verified the controller logic is correct (passed in our run). | **No fix needed at this slot.** Same shape as T3.6 — downstream of upstream WR signal being below the controller's design threshold. Will defrost automatically when WR climbs above 0.50. |
| T3.5 | `hold_cost_scale` | 0.0100 (at min floor) | Controller-driven within [min, max] bounds | Controller reads `hold_pct_ema` (which is also frozen at 0) — chain failure: T3.2 must fix first |
| ~~T3.6~~ | ~~`aux_conf_threshold`~~ | ~~0.0100 (at min floor)~~ | **WITHDRAWN 2026-05-10 (not a bug)**: investigation shows the controller at `sp20_controllers_compute_kernel.cu:152-157` (`clamp(aux_dir_acc_ema - 0.50, 0.01, 0.20)`) is *working correctly*. With observed `aux_dir_acc ≈ 0.46` (below random in d7bj7 logs), raw = -0.04, controller clamps at floor 0.01. Chain `aux_dir_acc_reduce_kernel → aggregator → EMA blend → controller` is intact and the EMA tracks the actual ~0.46 value. The slot will defrost on its own when aux improves above 0.51. **Downstream symptom of the upstream aux signal problem, NOT an independent producer-chain bug.** | **No fix needed at this slot.** Upstream problem (aux_dir_acc < 0.51) addressed by upstream feature/architecture work — separate from SP21 scope. |

**Strategy**: T3.1 fixed first as canonical example (it has a recent pearl + recent commit `64bbbe418` claiming the fix). Verify the patch actually propagated to the ISV slot, not just to `is_win_per_env` buffer. Once T3.1 is verified working, use the same pattern (find producer → check input availability → check write-to-isv → check sentinel guard) for T3.2-T3.6 in parallel.

**Success criterion**: HEALTH_DIAG `sp20_isv` line shows non-zero, varying values for all 6 slots across consecutive epochs. ISV slot variance > 0 over a 5-epoch window.

**Pearl compliance**:
- `feedback_isv_for_adaptive_bounds` — bounds must adapt
- `pearl_first_observation_bootstrap` — sentinel must be replaced on first valid observation
- `pearl_wiener_optimal_adaptive_alpha` — EMAs use Wiener-optimal α where applicable
- `pearl_blend_formulas_must_have_permanent_floor` — but only AFTER first observation; sentinel is not a floor

---

### Tier 4 — Controller-chain non-adaptation

Five HEALTH_DIAG controller readouts in `xmd6b` showing structural non-adaptation:

| # | Controller | Observed | Expected | Diagnosis |
|---|---|---|---|---|
| T4.1 | `c51_budget_per_branch` | all 0.0000 | per-branch C51 loss budget ∈ [0, 1] adapting on flatness per `pearl_per_branch_loss_budget` | Controller likely never producing any non-zero budget — input signal (Q-flatness) might be zero |
| T4.2 | `cql_budget_per_branch` | all 0.0000 | per-branch CQL loss budget | Same as T4.1 — controller chain dead |
| T4.3 | `q_var_per_branch` | all 0.0000 | per-branch Q-value variance | Either Q-variance producer dead, or all branches producing identical values |
| T4.4 | `iqn_budget_per_branch` | all 1.0000 (max) | adaptive IQN τ-schedule per `pearl_per_branch_iqn_tau_schedule` | Controller stuck at saturation |
| T4.5 | `lb_active_per_branch` | all 1.0000 | per-branch loss-balance active flag — should reflect per-branch loss budget | Likely synthesized from T4.1/T4.2 — fixes there might cascade |

**Strategy**: After T3 defrosts the foundational ISV slots, re-observe T4 readouts. T4 controllers are likely DOWNSTREAM of T3 inputs — fixing the source ISVs may cascade-fix the controllers. If not, audit each controller individually using the same pattern as T3.

**Success criterion**: HEALTH_DIAG controller readouts show non-trivial inter-branch variation across epochs.

**Pearl compliance**: `pearl_per_branch_c51_atom_span`, `pearl_per_branch_iqn_tau_schedule`, `pearl_per_branch_loss_budget`, `pearl_per_branch_noisy_sigma` all apply.

---

### Tier 5 — Design issues (not bugs)

| # | Finding | Implication | Disposition |
|---|---|---|---|
| T5.1 | Train uses Thompson + ε=0.05; val uses pure greedy (ε=0) | Direct metric comparison is apples-to-oranges | Per `pearl_thompson_for_distributional_action_selection.md`, both should use Thompson. **Decision**: align val to Thompson per the pearl, OR document the explicit deviation if greedy-val is required for some other reason. Recommendation: **align** — measuring the deployed policy under its true action distribution. |
| T5.2 | `epoch_sharpe` is per-trade Sharpe × √(trades_per_year), inflating values to [-90, +90] from raw values [-0.05, +0.05] | The metric we've been printing as the headline number is structurally meaningless | Either: (a) print `sharpe_raw` as the headline; (b) reduce trades_per_year annualization to actual trading days; (c) keep both, label clearly. Recommendation: **(c)** — keep both for backward-compat with archived runs, but make `sharpe_raw` the early-stop and best-checkpoint signal. |
| T5.3 | Hardcoded `adx_threshold = 30.0` for regime detection | `training_loop.rs:2187` | Magic number for ADX. ADX is a derived feature; its meaningful threshold depends on the feature's distribution at the current bar resolution. | Drive from `ISV[ADX_DISTRIBUTION_QUANTILE]` or equivalent — should mark "high ADX" as e.g. 75th percentile of observed ADX EMA, not absolute 30. If no producer exists, T5.3 grows to include adding one. |

**Pearl compliance**: T5.1 honors `pearl_thompson_for_distributional_action_selection`.

---

## Sequencing

Per `feedback_no_deferrals_for_complementary_fixes.md`, all findings are scoped into one superproject. But execution is per-tier with clear gates so a regression at any tier doesn't poison subsequent tiers:

1. **Tier 1 first** (smaller-blast-radius fixes):
   - T1.1a + T1.1b atomic (early stop swap source AND signal-drive min_delta + update tests)
   - T1.2 (wire real values to enrichment caller — observations only)
   - T1.4 (signal-drive backtracking thresholds)
   - **T1.3 deferred to T2.2** — see below
   Verify: training run shows early-stop near val_Sharpe peak; backtracking saves at correct cadence; enrichment caller receives real values (add a one-time log line proving it).

2. **Tier 2 architectural debt** — execute T2.1 (delete avg_q_value path), T2.2+T1.3 jointly (rewrite enrichment.rs as ISV-aware functions). T2.3 is part of T1.1b's atomic commit.
   Verify: enrichment outputs differ from previous run (proving it's now signal-driven); early-stop and best-ckpt selection use only val_loss path.

3. **Tier 3 ISV defrost**, T3.1 first as canonical example, then T3.2-T3.6 in parallel using the same template (find producer → check input availability → check write-to-isv → check sentinel guard).
   Verify: HEALTH_DIAG `sp20_isv` line shows variance > 0 across consecutive epochs for ALL 6 frozen slots.

4. **Tier 4 controller cascade** — re-check after Tier 3 fix; investigate residuals. Many T4 controllers may unfreeze automatically once their T3 inputs unfreeze (cascade hypothesis).
   Verify: HEALTH_DIAG controller readouts show inter-branch variation.

5. **Tier 5 design alignments** — Thompson-in-val and metric clarity (T5.1, T5.2), regime threshold ISV-driving (T5.3). Larger scope; may be split into a separate SP if T5.3 requires building a new ADX EMA producer.

Each tier completion = one commit (or commit set), with smoke test verification before proceeding to next tier. **Tier 1 has roughly 100-200 lines of code change**; **Tier 2+T1.3 has ~400 lines** (enrichment rewrite); **Tier 3 has ~50 lines per slot × 6 slots = ~300 lines**; **Tier 4-5 sized after Tier 3 results.**

---

## Open questions before execution

1. **T1.1 design**: T1.1a (swap source) is a 1-line change. T1.1b (signal-driven min_delta) is a refactor of `EarlyStopping::should_stop` to accept per-call `min_delta`. Two sub-questions:
   (a) On the very first call when `ISV[VAL_SHARPE_VAR_EMA]=0` (sentinel), what permanent floor for `min_delta`? Per `pearl_blend_formulas_must_have_permanent_floor`, suggest `max(1e-3, k × sqrt(var_ema))` with `k=1.0`.
   (b) Multiplier `k` — 1σ improvement (k=1) vs 0.5σ (k=0.5) vs 2σ (k=2). Higher k = stricter "real improvement" threshold, fewer false-positive resets, more false-negative early-stops. Default k=1.0 as starting point but **k itself could be ISV-driven from val_sharpe trend stability** if we want fully adaptive — that's a sub-pearl decision.
2. **T2.2 scope**: refactoring `extract_eval_trades_from_metrics` away requires touching every enrichment function. How many call sites? Risk of partial refactor per `feedback_no_partial_refactor`.
3. **T3 producer locations**: ISV slot indices are known (`AUX_PRED_HORIZON_BARS_INDEX=450`, etc.) but the producer kernels for each frozen slot need to be located. Some are pearl-named; others may not be wired to anything.
4. **T5.1 implication**: aligning val to Thompson may *change* the val_Sharpe numbers we've been seeing. Pre/post comparison matters — we don't want to "fix" a metric and then look like the policy degraded when it actually didn't.

---

## Success criteria — superproject level

The whole SP21 is complete when, on a fresh training run:

1. `🛑 Early stopping triggered` fires within `patience` epochs of val_Sharpe peak
2. HEALTH_DIAG `sp20_isv` line shows variance > 0 across epochs for ALL 6 frozen slots
3. HEALTH_DIAG controller readouts (`c51_budget`, `cql_budget`, `q_var`, `iqn_budget`, `lb_active`) show inter-branch variation
4. Enrichment log line proves real values are flowing in
5. Train/val metrics are coherent in DIRECTION (both rise OR both fall during a learning epoch; not opposite signs)
6. `evaluate_baseline` (or equivalent eval entry point) reproduces the val_Sharpe / val_PF observed in training (within 5% relative tolerance) when run on the same checkpoint + window

If criterion 6 fails, there's a hidden train/val pipeline divergence we haven't surfaced — open a SP22 to investigate.

---

## Non-goals

- Action-space redesign (Hold/Flat split, magnitude bucketing) — separate SP
- Feature pipeline changes (MBP10, OFI, TLOB, imbalance bar threshold tuning) — separate SP
- Reward shape redesign (asymmetric clamp, opportunity cost, micro-reward weights) — separate SP
- Hyperparameter sweep — moot until coherence is restored

---

## Pearls created or amended by this superproject

Likely candidates (to formalize after execution):

- New pearl: `pearl_validate_metric_inputs_not_just_outputs` — periodically verify that EMA/controller producers are receiving non-trivial inputs, not just that outputs change. Frozen `wr_ema=0` is what `is_win_per_env=0` always looks like — both are silent failures.
- New pearl (or amend `feedback_no_todo_fixme`): hardcoded placeholder constants in production paths must be flagged at compile time, not runtime.
- Possibly amend `pearl_first_observation_bootstrap`: sentinel-detection thresholds must match the producer kernel's actual sentinel values (T3.4 `loss_cap=-1` may have a guard checking for `0` or `NaN` instead of `-1`).
