# SP11 — Reward as Controlled Subsystem (fully ISV-driven)

**Date:** 2026-05-04
**Status:** Draft (user review pending)
**Authors:** session-1777830566225
**Related pearls:**
- `pearl_controller_anchors_isv_driven.md` (every threshold ISV-driven)
- `pearl_cold_start_exit_signal_or.md` (OR'd canary signals)
- `pearl_blend_formulas_must_have_permanent_floor.md` (floors prevent zero-out)
- `pearl_engagement_rate_self_correction.md` (saboteur engagement self-corrects)
- `pearl_wiener_optimal_adaptive_alpha.md` (Pearl D — α from variance ratio)
- `pearl_first_observation_bootstrap.md` (Pearl A sentinel-bootstrap)

## 1. Overview

The Fix 33-38 chain made every CONTROLLER signal-driven (CQL budget, MAX_BUDGET,
Kelly cap, Thompson temperature). The reward subsystem is the last
regime-encoded layer in the training stack. SP11 promotes every reward
component weight, shaping multiplier, exploration bonus, and curriculum knob
to ISV-driven slots. **Every input is itself ISV-driven** — no hardcoded
threshold, no constant floor that isn't an Invariant-1 numerical anchor.

The new pearl this authors (`pearl_reward_as_controlled_subsystem`) declares
the principle: every degree of freedom in the reward path is an output of a
unified controller whose inputs are improvement-rate canaries.

## 2. Pathology

T10 train-multi-seed-xkjkb seed-0 ep0-14 evidence (commit `920a2d021`,
all SP10 + earlier fixes applied):

```
ep0:  val sharpe=75.18 trade_count=134327 active_frac=0.502 dir_entropy=1.041
ep1:  val sharpe=80.61 trade_count=133997 active_frac=0.500 dir_entropy=1.039  ← peak
ep2:  val sharpe=77.64 trade_count=131911 active_frac=0.481 dir_entropy=1.025
...
ep10: val sharpe=73.10 trade_count=120617 active_frac=0.403 dir_entropy=0.953
ep14: val sharpe=70.58 trade_count=117236 active_frac=0.384 dir_entropy=0.932

train: Q-value 0.04 → 0.30 (growing), train_loss stable 3.7-4.1
```

**Diagnosis:** the model finds a stable fixed point at ep1 then OVERFITS to
it. Q-values grow (TD targets propagate) but val performance declines because
the policy has nothing pushing it to be _better_, only to be _consistent_.
The reward function is purely "how good was this bar" with no improvement
pressure, no curriculum gradient, no curiosity.

The reward components (`popart`, `cf`, `trail_r`, `micro`, `opp_cost`,
`bonus`) all have hardcoded weights (`cf_weight = 0.3f` at
`mse_loss_kernel.cu:318` and `c51_loss_kernel.cu:789`; saboteur intensity
decays on a hardcoded 0.995/epoch schedule in `gpu_experience_collector.rs`).
None adapt to whether training is improving, stagnating, or regressing.

### 2.1 Design awareness — Q-overconfidence is also in scope

The "Q grows while val drops" signature can also be driven by
**TD-target overconfidence**: optimistic bootstrap targets pull the
value function toward higher magnitudes regardless of policy quality.
SP11's CQL component-weight slot (`REWARD_CF_WEIGHT_INDEX`) explicitly
covers this case: when `improving_z < 0` (regressing), the
controller diversifies weights toward under-used components, which
includes raising CQL conservatism if the grad-ratio canary shows CQL
under-firing. Because the controller's input space includes the very
shaping multipliers that govern Q-target conservatism, SP11 *is* the
fix for both reward-shaping and conservatism in one architecture.

No separate SP11′ is needed. If T10 (§9) shows residual decline after
SP11, the response is to extend the controller's outputs further (e.g.,
target-update τ as a new ISV slot), not to pause SP11.

## 3. Architecture (refined: every input ISV-driven)

### 3.1 Z-score-driven adaptation (no hardcoded thresholds)

The "improving / stagnant / regressing" classification uses a **true Z-score**
(mean Δ divided by std of residuals around the mean — *not* mean over
magnitude, which would bound z to [-1, +1] and prevent sigmoid saturation,
collapsing the controller's effective range to [0.27, 0.73]):

```
delta             = val_sharpe[t] − val_sharpe[t-1]
delta_ema         = EMA(delta)                          ← running mean of Δ
delta_var_ema     = EMA( (delta − delta_ema)² )         ← variance around the trend
improvement_z     = delta_ema / max( sqrt(delta_var_ema), EPS_DIV )
```

`improvement_z` is unbounded so `sigmoid(z)` can saturate near 0 or 1 in
regimes that warrant it. Both EMAs are Pearl D Wiener-α-smoothed and
Pearl A sentinel-bootstrapped (first observation replaces sentinel
directly). The variance EMA is computed inside the canary kernel in a
two-pass form: first update `delta_ema`, then accumulate the squared
deviation against the *updated* mean (corrects for the EMA-of-EMA bias).

`improvement_z` ∈ ℝ; the controller's outputs are continuous functions of
`z`, not threshold-gated:
- `z > 0`: improving — exploit current, raise curriculum, reweight toward
  contributing components
- `z ≈ 0`: stagnant — raise curiosity, raise saboteur intensity, diversify
  weights
- `z < 0`: regressing — max curiosity, lower saboteur (relax curriculum),
  return weights to floor mean

### 3.2 Outputs (10 ISV slots)

| Slot | Purpose | Floor |
|---|---|---|
| `REWARD_POPART_WEIGHT_INDEX` | PnL-magnitude reward weight | signal-driven |
| `REWARD_CF_WEIGHT_INDEX` | Counterfactual reward weight | signal-driven |
| `REWARD_TRAIL_WEIGHT_INDEX` | Trailing-stop reward weight | signal-driven |
| `REWARD_MICRO_WEIGHT_INDEX` | OFI micro-structure reward weight | signal-driven |
| `REWARD_OPP_COST_WEIGHT_INDEX` | Flat opportunity cost weight | signal-driven |
| `REWARD_BONUS_WEIGHT_INDEX` | Trade-attempt/timing bonus weight | signal-driven |
| `CURIOSITY_PRESSURE_INDEX` | NEW exploration-bonus magnitude | bounded by `pnl_reward_magnitude_ema` |
| `SABOTEUR_INTENSITY_MULT_INDEX` | Curriculum modulator on saboteur scale | Invariant-1 [0.5, 2.0] |
| `REWARD_WEIGHT_FLOOR_INDEX` | Per-component minimum weight | signal-driven from grad-history |
| `CURIOSITY_BOUND_INDEX` | Max curiosity bonus magnitude | signal-driven from PnL EMA |

### 3.3 Canaries (5 ISV slots)

| Slot | Source | Pearl pattern |
|---|---|---|
| `VAL_SHARPE_DELTA_EMA_INDEX` | `Δsharpe[t] = sharpe[t]−sharpe[t-1]`, Pearl D smoothed | A+D |
| `VAL_SHARPE_VAR_EMA_INDEX` | EMA of `(Δsharpe − delta_ema)²` (true variance estimator) | A+D |
| `REWARD_COMPONENT_GRAD_RATIO_BASE` | **6 slots** [base..base+6), per-component grad-EMA / total-grad-EMA | A+D |
| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Per-bar saboteur-driven Q-shift hit rate (see §3.3.1 for operational definition) | Pearl C engagement counter |
| `PNL_REWARD_MAGNITUDE_EMA_INDEX` | `|popart_reward|` EMA — bounds curiosity proportionally | A+D |

#### 3.3.1 Operational definitions

**Saboteur engagement** (Foxhunt has bars, not episodes): inside
`gpu_experience_collector`, the saboteur perturbs fill price and slippage
per-bar. Engagement is the fraction of bars where the perturbation
*actually changed the realized reward by ≥ EPS_ENGAGEMENT*:
```cuda
bool engaged = fabsf(reward_with_saboteur - reward_without_saboteur) > EPS_ENGAGEMENT;
```
The "without saboteur" reward is computed cheaply by un-applying the
perturbation (price + perturbation_offset → price). This is a per-bar
boolean reduced via block tree-reduce (no atomicAdd, per
`feedback_no_atomicadd`) into a per-step engagement count, then divided
by batch size and EMA'd. EPS_ENGAGEMENT = `0.01 × isv[PNL_REWARD_MAGNITUDE_EMA]`
(Invariant-1 fraction of PnL signal scale).

**Per-component grad ratio**: each loss kernel already accumulates
per-component gradient magnitudes for the SP4 grad-balancer. Reuse those
6 EMAs; ratio = `grad_ema[c] / max(sum(grad_ema), EPS_DIV)`.

### 3.4 Producer kernel: `reward_subsystem_controller`

Single block, 10 threads (one per output). Reads all 5 canaries, computes:

```cuda
const float EPS_DIV = 1e-6f;
const float SABOTEUR_MIN = 0.5f;          // Invariant-1 rate limiter
const float SABOTEUR_MAX = 2.0f;          // Invariant-1 rate limiter
const float WEIGHT_HARD_FLOOR = 0.01f;    // Invariant-1 numerical floor
const float CURIOSITY_PERMANENT_FRACTION = 0.2f;  // Invariant-1 floor fraction
const float ENGAGEMENT_FLOOR = 0.1f;      // Invariant-1 fraction
const float CURIOSITY_BOUND_FRACTION = 0.3f;       // Invariant-1 fraction of PnL EMA
const float WEIGHT_FLOOR_FRACTION = 0.5f;          // Invariant-1 fraction of min grad ratio

// Block-load all 6 ratios into shared mem ONCE (avoids re-reading ISV).
__shared__ float ratios[6];
if (threadIdx.x < 6) {
    ratios[threadIdx.x] = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + threadIdx.x];
}
__syncthreads();

// True Z-score: mean Δ over std-of-residuals, NOT mean Δ over mean |Δ|.
// (Latter would be bounded to [-1,+1] and never let sigmoid saturate.)
float delta_var = isv[VAL_SHARPE_VAR_EMA_INDEX];
float dz = isv[VAL_SHARPE_DELTA_EMA_INDEX]
         / fmaxf( sqrtf(delta_var), EPS_DIV );

// Sigmoid maps z continuously to [0, 1]
float improving = 1.0f / (1.0f + expf(-dz));    // 0=regressing, 0.5=stagnant, 1=improving
float stagnant_or_worse = 1.0f - improving;

// Per-component weight FLOOR (signal-driven from worst-suppressed component)
float min_grad_ratio = ratios[0];
for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]);
float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, WEIGHT_FLOOR_FRACTION * min_grad_ratio);
scratch[scratch_weight_floor] = adaptive_floor;

// Per-component weights — blend pre-floor, then enforce floor, then RENORM.
// Renormalization is critical: PopArt's reward-magnitude EMA assumes a
// consistent reward-scale; sum(weights) drifting > 1 would inflate the
// running |reward| and contaminate the popart target every fold.
float blends[6];
float blend_sum = 0.0f;
for (int c = 0; c < 6; c++) {
    float winner_weight = ratios[c];                    // exploit
    float diversifier_weight = (1.0f - ratios[c]) / 5.0f; // explore (sums to 1−ratios[c]·5/5 across c, see note)
    float w = improving * winner_weight + stagnant_or_worse * diversifier_weight;
    w = fmaxf(adaptive_floor, fminf(1.0f, w));
    blends[c] = w;
    blend_sum += w;
}
// Renormalize so mean(weights) = 1 (Σ = N = 6). EPS_DIV guards the
// (impossible) all-floor-zero case. **Amended 2026-05-04 — see §3.4.3
// for rationale (the original A2-landed code used Σ = 1, which collapses
// reward magnitude 6× on mutually-exclusive components).**
const float N_components = 6.0f;
float scale = N_components / fmaxf(blend_sum, EPS_DIV);
for (int c = 0; c < 6; c++) {
    scratch[scratch_weight_base + c] = fminf(MAX_WEIGHT, blends[c] * scale);
}

// Curiosity bound = fraction of recent PnL magnitude (signal-relative)
float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA_INDEX] * CURIOSITY_BOUND_FRACTION;
scratch[scratch_curiosity_bound] = curiosity_bound;

// Curiosity pressure: ALWAYS-ON exploration signal scaled by stagnation.
//   When improving: floor preserves baseline exploration (anti-overfitting).
//   When stagnating: rises toward curiosity_bound to break out of fixed point.
// Permanent floor pattern per pearl_blend_formulas_must_have_permanent_floor —
// max() form, not blend, so curiosity never zeros even when policy is winning.
float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound;
float curiosity_dynamic = stagnant_or_worse * curiosity_bound;
float curiosity_pressure = fmaxf(curiosity_dynamic, curiosity_floor);
scratch[scratch_curiosity_pressure] = curiosity_pressure;

// Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX]
// modulated by engagement rate (per pearl_engagement_rate_self_correction).
// Compute the z-term inside the bound, modulate by engagement, and re-clamp
// to [SABOTEUR_MIN, SABOTEUR_MAX]. Without the post-clamp the engagement
// floor (0.1) would silently pull the output below SABOTEUR_MIN.
float engagement = isv[SABOTEUR_ENGAGEMENT_RATE_INDEX];
float intensity_z_term = SABOTEUR_MIN + (SABOTEUR_MAX - SABOTEUR_MIN) * improving;
float modulated = intensity_z_term * fmaxf(engagement, ENGAGEMENT_FLOOR);
float saboteur_intensity = fmaxf(SABOTEUR_MIN, fminf(SABOTEUR_MAX, modulated));
scratch[scratch_saboteur_intensity] = saboteur_intensity;
```

#### 3.4.1 Output smoothing (Pearls A+D applied to controller outputs)

The 10 controller outputs are written to `scratch_*` first, then a chained
`apply_pearls_ad_kernel` launch smooths each output into its ISV slot.
This prevents the controller from oscillating outputs at every training
step (raw `improving` is responsive but jittery). Pearl A handles the
fold-boundary cold-start; Pearl D's Wiener-α adapts smoothing strength to
each output's own noise level (curiosity may be smoother than weights, etc.)

This means the consumer kernels read the **smoothed** controller outputs
from ISV, not the raw scratch values.

#### 3.4.2 Surviving constants (Invariant-1 anchors only)

All seven hardcoded constants in the controller bound *rates*, not *regimes*,
and remain valid across regime shifts:

| Const | Value | Role |
|---|---|---|
| `EPS_DIV` | `1e-6f` | divide-by-zero protection |
| `WEIGHT_HARD_FLOOR` | `0.01f` | absolute weight floor — prevents zero |
| `MAX_WEIGHT` | `3.0f` | per-component upper cap (mean=1; cap = 3× over-weighted) |
| `SABOTEUR_MIN` / `MAX` | `0.5f` / `2.0f` | curriculum rate limiter |
| `CURIOSITY_PERMANENT_FRACTION` | `0.2f` | exploration floor as fraction of `curiosity_bound` |
| `CURIOSITY_BOUND_FRACTION` | `0.3f` | curiosity ceiling as fraction of PnL EMA |
| `WEIGHT_FLOOR_FRACTION` | `0.5f` | adaptive floor as fraction of min grad ratio |
| `ENGAGEMENT_FLOOR` | `0.1f` | saboteur engagement floor (pre-bound) |

The thresholds that classify regime ("improving / stagnant / regressing")
are entirely encoded in the Z-score and sigmoid — no hardcoded threshold
survives at the regime layer.

#### 3.4.3 Mean-normalization (amendment 2026-05-04 post-A2 audit)

**A2-landed code uses `Σweights = 1`.** Implementer audit during B1 surfaced
that `experience_env_step` reward composition is *mutually-exclusive
inline accumulation* (popart / micro / opp_cost are alternative paths;
exactly one fires per bar), so `Σ w_i × r_i` reduces to `single_weight ×
single_value`. With Σ=1 across 6 components, the average per-bar weight
is ~1/6 → 6× collapse of reward magnitude vs pre-SP11. PopArt EMA tracks
but adapts on a multi-epoch timeconstant, interacting with C51 v_range /
IQN τ adaptation and risking cold-start training pathology.

**Amendment:** weights are normalized to **mean = 1.0** (i.e., `Σ = N
where N = 6`), not `Σ = 1`. Each weight ∈ [floor, MAX_WEIGHT] where
`MAX_WEIGHT = 3.0` (Invariant-1 cap preventing any single weight
dominating ≥ 50% of the 6-component budget). Default uniform = 1.0
each. The controller's renormalization step in §3.4 changes from:

```cuda
inv_sum = 1.0 / max(blend_sum, EPS_DIV);
scratch[c] = blends[c] * inv_sum;          // Σ = 1
```

to:

```cuda
scale = (float)N / max(blend_sum, EPS_DIV);
scratch[c] = fminf(MAX_WEIGHT, blends[c] * scale);   // mean = 1
```

This preserves pre-SP11 absolute scale on average while letting the
controller redistribute share among components. PopArt's `popart` slot
weight (default 1.0) acts as a multiplicative around 1.0× of pre-SP11
trade-PnL, not a 1/6 attenuation.

§9.1 smoke pass criterion changes from `Σweights ≈ 1.0 ± 1e-3` to
`mean(weights) ≈ 1.0 ± 1e-3` (i.e., Σ ≈ 6.0 ± 6e-3 with N=6 components).

#### 3.4.4 Universal post-composition modifiers (amendment 2026-05-04)

Six modifiers in `experience_env_step` apply AFTER the if/else reward
composition: drawdown penalty, capital-floor cap, inventory penalty,
churn penalty, conviction-scale multiplier, G7 counterfactual flip.
**These are RISK CONSTRAINTS or STRUCTURAL OPERATORS, not reward
components** — they don't get controller weights. Concretely:

- **Drawdown / inventory / churn penalties** are unconditional risk
  guards that fire on every bar matching their condition. The whole
  point is that the agent CANNOT learn to weigh them away. They apply
  post-composition, unweighted.
- **Capital-floor cap** is a hard clamp `r = max(-10, -5 - 5×risk_taken)`
  on capital breach — a structural safety bound, not a weighted
  contribution.
- **Plan conviction scale** is a multiplier `r *= conviction` driven by
  the plan head's confidence — modulates the existing reward, not a
  separate component to weight.
- **G7 cf-flip** is `r *= -1` on counterfactual samples — a structural
  operator on the cf path, not a learning weight.

Composition with universal modifiers:

```
r_weighted = Σ w_i × r_i                     // SP11 controller-weighted
r_total    = clamp_capital(r_weighted)       // hard clamp
r_total   -= drawdown_penalty                // unconditional risk guard
r_total   -= inventory_penalty               // unconditional risk guard
r_total   -= churn_penalty                   // unconditional risk guard
r_total   *= conviction_scale                // plan modulator
r_total   *= cf_flip                         // structural operator
out_rewards[out_off] = r_total
```

The controller's 6 weights govern WHICH learning signals matter
(`Σ w_i × r_i`). The post-composition modifiers govern what the model
must always pay attention to regardless of the controller. Both are
correct separately; no controller weights for the modifiers.

### 3.5 Consumer migrations

**`REWARD_CF_WEIGHT_INDEX` scope clarification (amendment 2026-05-04):**
The `cf_weight = 0.3f` constants at `mse_loss_kernel.cu:318` and
`c51_loss_kernel.cu:789` are **structural Q-distribution mixing weights
for Hold-sample magnitude branch gradient**, NOT reward-shaping weights.
They happen to share the variable name with the cf-reward path but are a
DIFFERENT signal entirely. The original draft of this spec listed them
for migration; that was an audit error. Leave both untouched.

`REWARD_CF_WEIGHT_INDEX` applies ONLY to the cf-reward path inside
`experience_env_step` — the counterfactual-sample reward written to
`out_rewards[cf_off]` and consumed by the loss kernels via the replay
buffer's cf tuples. The structural-refactor composition `Σ w_i × r_i`
applies `w_cf × r_cf` for cf samples; on-policy samples use
`w_popart × r_popart` (or whichever path fired) without crossing the cf
weight at all.

**Reward composition site** (the single migration target — see §3.5.3
for the structural decomposition):
- `experience_env_step` reward composition: 8+ inline accumulation sites
  decomposed into per-component locals, then composed via the weighted
  Σ formula in §3.4.4 (with `mean(weights) = 1` normalization per §3.4.3
  and post-composition risk modifiers per §3.4.4).

**Curiosity bonus** (NEW reward dimension, applied at *loss-compute* time
not at experience-collect time — see §3.5.1):
- The replay buffer continues to store the **base reward** only (no
  curiosity baked in). Curiosity is added to `r` at the moment a tuple is
  drawn for replay, recomputed against the *current* visit-count table
  and *current* `CURIOSITY_PRESSURE_INDEX`.
- Recompute formula: `r_with_curiosity = r_base + isv[CURIOSITY_PRESSURE_INDEX]
  * novelty_signal(state, action)`
- Bounded above by `CURIOSITY_BOUND_INDEX` in the producer kernel.

#### 3.5.3 Structural decomposition of `experience_env_step` reward composition (amendment 2026-05-04)

The pre-SP11 `experience_env_step` accumulates reward in-place across
8+ inline sites (search audit identified `:2587` trade-PnL base, `:2845`
micro overwrite, `:2963` Flat opp-cost, `:2646/:2667/:2718/:3011/:3061`
bonus accumulations, `:3400` cf_reward — line numbers as of 2026-05-04
audit; verify at implementation time). The migration replaces this with
explicit per-component locals + a final weighted sum:

```cuda
float r_popart   = 0.0f;   // trade-PnL on segment-complete bars
float r_cf       = 0.0f;   // counterfactual-sample reward
float r_trail    = 0.0f;   // trail-fire P&L (§3.5.4)
float r_micro    = 0.0f;   // OFI-driven micro-reward (mid-trade)
float r_opp_cost = 0.0f;   // Flat opportunity cost
float r_bonus    = 0.0f;   // B/C/D bonuses summed

// ... existing per-bar logic populates ONE OR MORE of the locals ...

const float w_popart   = isv[REWARD_POPART_WEIGHT_INDEX];
const float w_cf       = isv[REWARD_CF_WEIGHT_INDEX];
const float w_trail    = isv[REWARD_TRAIL_WEIGHT_INDEX];
const float w_micro    = isv[REWARD_MICRO_WEIGHT_INDEX];
const float w_opp_cost = isv[REWARD_OPP_COST_WEIGHT_INDEX];
const float w_bonus    = isv[REWARD_BONUS_WEIGHT_INDEX];

float r_weighted = w_popart   * r_popart
                 + w_cf       * r_cf
                 + w_trail    * r_trail
                 + w_micro    * r_micro
                 + w_opp_cost * r_opp_cost
                 + w_bonus    * r_bonus;

// Universal post-composition modifiers (§3.4.4):
float r_total = clamp_capital(r_weighted);
r_total -= drawdown_penalty;
r_total -= inventory_penalty;
r_total -= churn_penalty;
r_total *= conviction_scale;
r_total *= cf_flip;

total_reward_per_sample[out_off] = r_total;
```

**Mutual-exclusivity preservation:** popart / micro / opp_cost are
mutually exclusive paths in the if/else chain. The migration writes 0
to the locals that don't fire and the relevant local to the active
component. The weighted sum produces `w_active × r_active` per bar (one
multiply, not 6), which with `mean(w) = 1` (§3.4.3) preserves the
pre-SP11 absolute scale.

#### 3.5.4 Trail reward extraction (amendment 2026-05-04)

In pre-SP11 code, trail-fire P&L flows through `segment_complete = true`
because `exiting_trade = trail_triggered ? 1 : wants_exit` — both
voluntary-exit and trail-forced-exit go through the same popart-credit
path. So `r_trail` would be structurally inert (always 0) under a naive
decomposition, leaving `REWARD_TRAIL_WEIGHT_INDEX` as a dead controller
output.

**Architectural change:** segregate trail-fire bars from voluntary-exit
bars in `segment_complete`'s P&L credit:

```cuda
if (segment_complete) {
    const float pnl = computed_segment_pnl;
    if (trail_triggered) {
        r_trail = pnl;     // forced-exit P&L → trail signal
    } else {
        r_popart = pnl;    // voluntary-exit P&L → popart signal
    }
    // ... existing segment_complete bonus accumulations ...
}
```

This gives the controller meaningful signal on `w_trail` to weigh
forced-exit P&L differently from voluntary-exit P&L (e.g., upweight
forced-exit when trail-stop fires too aggressively, downweight when
trail-stop is too loose). The `pearl_trail_fire_pre_vs_action_mag`
already documents that trail filtering uses prior-position magnitude;
this extension lets the controller learn whether the trail-stop is
producing too much/too little forced-exit signal.

#### 3.5.5 Replay-time curiosity audit gate (amendment 2026-05-04)

The B1c (curiosity wiring) commit depends on a verifiable claim: the
trainer's `rewards_buf` is the SINGLE read site for replay rewards
across ALL downstream consumers (mse_loss, c51_loss, IQN target, CQL,
Bellman target). If any consumer reads from a different buffer (e.g., a
target-specific reward buffer populated separately), post-gather
curiosity-add doesn't propagate to that consumer, creating a partial
migration violating `feedback_no_partial_refactor`.

**Layer C audit task (precedes B1c commit):**
1. `grep -rn "rewards_buf\|rewards_dev\|reward_target" crates/ml/src/cuda_pipeline/`
2. Trace each non-write read; document the consumer + which buffer it
   reads.
3. Verdict matrix: { consumer → which buffer it reads }. Pass criterion:
   every consumer reads from `trainer.rewards_buf` (the post-gather
   buffer that B1c augments with curiosity). If any consumer reads
   elsewhere, B1c either (a) extends to that buffer, or (b) the spec
   is amended to acknowledge that consumer is intentionally
   curiosity-free.

#### 3.5.1 Why recompute-at-replay (not store-at-collect)

If we store curiosity-augmented r in replay, old tuples retain the
curiosity bonus they had *at the moment they were collected*, even though
the policy now visits those state-actions frequently (curiosity should be
near zero). Replaying those tuples then teaches the policy to revisit the
fixed point we're trying to escape — making the ep1-overfitting pathology
worse, not better. Recomputing at replay binds curiosity to the *current*
exploration deficit, which is the only signal that matters.

The cost is one extra hash-lookup + multiply per replayed tuple. Existing
replay sampling kernel adds two ops; no extra kernel launch.

#### 3.5.2 Novelty signal (state-action visit count via locality-sensitive hash)

The state space is continuous 42-dim. Use a SimHash-style discretization:
- Project state through a fixed (untrained) random matrix `W ∈ R^{42×16}`
  with sign-quantized output: `code = sign(state @ W)` → 16-bit hash.
- Bucket = `(action_id, code)` — i.e., 108 actions × 65,536 codes = ~7M buckets.
- Visit count = atomic-free per-bucket EMA in a hash table sized 1M slots
  (with linear-probe collisions tolerated; collisions = false novelty
  underestimates, which is the safe direction).
- `novelty_signal = 1 / sqrt(1 + count)`. Range [0, 1].
- Hash table is per-fold (cleared at fold boundary via state-reset registry).
- The 42×16 projection matrix is fixed at trainer init from a deterministic
  seed (so different runs with same seed get same hash function).

This is a cheap, GPU-friendly stand-in for RND or count-based bonuses. RND
or full state-action density estimation can replace it later if smoke
shows the hash discretization is too coarse.

**Saboteur intensity** (modulator on existing saboteur scale):
- `gpu_experience_collector.rs:3328` — `let ps =
  self.saboteur_perturbation_scale * isv[SABOTEUR_INTENSITY_MULT_INDEX]` (was
  raw `saboteur_perturbation_scale`)
- The existing 0.995/epoch decay schedule remains as the BASE; the ISV
  multiplier is the adaptive _curriculum modulator_ on top

### 3.6 Reset semantics

All **20** new slots get FoldReset entries (sentinel 0, Pearl A bootstraps
from first observation). Cross-fold persistence is wrong for every slot
in this spec because the "improvement signal" must reset at each fold's
cold-start — train history from fold N-1 is not informative about
fold N's improvement trajectory.

This includes:
- 6 component weights (340..346)
- 4 controller scalar outputs (346..350)
- 2 val-sharpe canaries (350..352)
- 6 grad-ratio canaries (352..358)
- 1 saboteur engagement (358)
- 1 PnL magnitude EMA (359)

**Cold-start window:** The Z-score is undefined until at least 2 val
emits exist within the fold (need delta + variance). For epoch 0 of each
fold, the controller observes `delta_ema = 0` (post-Pearl-A bootstrap),
which gives `dz = 0`, `improving = 0.5`, `stagnant_or_worse = 0.5` — the
"perfectly stagnant" mid-state. Every controller output is well-defined
in this state: weights blend 50/50, curiosity sits at midpoint, saboteur
sits at the geometric mean of [MIN, MAX]. There is no NaN path.

The **novelty hash table** (§3.5.2) also resets at fold boundary; this is
NOT an ISV slot but a separate device buffer cleared by the state-reset
dispatcher.

## 4. ISV slot allocation

After SP10's last slot at 339:

| Slot range | Purpose | Count |
|---|---|---|
| `[340..346)` | 6 component weights | 6 |
| `346` | `CURIOSITY_PRESSURE_INDEX` | 1 |
| `347` | `SABOTEUR_INTENSITY_MULT_INDEX` | 1 |
| `348` | `REWARD_WEIGHT_FLOOR_INDEX` (adaptive) | 1 |
| `349` | `CURIOSITY_BOUND_INDEX` (signal-relative) | 1 |
| `350` | `VAL_SHARPE_DELTA_EMA_INDEX` (canary) | 1 |
| `351` | `VAL_SHARPE_VAR_EMA_INDEX` (canary, true variance) | 1 |
| `[352..358)` | `REWARD_COMPONENT_GRAD_RATIO_BASE` (6 canaries) | 6 |
| `358` | `SABOTEUR_ENGAGEMENT_RATE_INDEX` (canary) | 1 |
| `359` | `PNL_REWARD_MAGNITUDE_EMA_INDEX` (canary) | 1 |
| `360` | `POPART_COMPONENT_MAG_EMA_INDEX` (canary, B1b fix-up) | 1 |
| `[361..367)` | `REWARD_COMPONENT_VAR_EMA_BASE` (NEW B1b smoke-recovery — 6 per-component variance EMAs for z-score normalization) | 6 |

Total: **27 new slots**. Bump `SP5_SLOT_END = 367`, `ISV_TOTAL_DIM = 367`.

**Why slots 361..367 (amendment 2026-05-04 post-B1b smoke):** smoke-test-6wd2c
on commit `61b2fa962` (B1b + 4 bug fix-ups) revealed a 5th pathology:
the SP11 mag-ratio canary uses linear magnitude ratios `r_i / Σ r_j`,
but reward components have **intrinsically asymmetric magnitudes**:

- popart (trade P&L on segment_complete): O(100) per fire
- cf, trail, micro, opp_cost, bonus: O(0.1–2) each

Even with the slot 360 fix preventing total-reward contamination,
popart's intrinsic magnitude makes `popart_mag / Σ ≈ 0.93` whenever
popart fires. Controller blend: `winner_weight = ratio` → high ratio →
controller amplifies popart further. Smoke trajectory:
`w_pop=2.0 → 2.44 → 2.57` (heading to MAX_WEIGHT cap), `curiosity_b
=30 → 120 → 199` (popart_mag_ema growing exponentially), `sharpe_ema
10.7 → 2.4 → 0.75` (collapse).

This is **a design flaw, not a code bug**. The "winner_weight = ratio"
mechanism amplifies whichever component is intrinsically biggest,
regardless of whether that's a useful signal to amplify. Per-component
gradient ratios (the original spec intent) don't fix this either — in
DQN's architecture there's no per-component gradient pathway; gradient
norm scales with `current_weight × magnitude`, so it's the same bias.

**Resolution: z-score normalization.** Each component's magnitude is
divided by its OWN running standard deviation before computing the
ratio:

```
popart_z = popart_mag_ema / max(sqrt(popart_var_ema), EPS_DIV)  ≈ O(1)
cf_z     = cf_mag_ema     / max(sqrt(cf_var_ema), EPS_DIV)      ≈ O(1)
... etc for all 6 components

sum_z    = Σ component_z
ratio[c] = component_z / sum_z   ≈ ~1/6 each when all components stable
```

Now the canary signals "this component is *unusually active relative to
its own history*" rather than "this component is bigger than others".
Controller's `winner_weight = ratio` amplifies whichever component is
*currently surprising*, not the intrinsically-largest. Statistically
principled (z-score is the standard scale-invariant measure of
significance) and matches what the SP11 controller is trying to
express.

The 6 new variance EMA slots [361..367) parallel slots [63..68) (mag
EMAs) + slot 360 (popart-component-specific mag EMA) but track
`var(|r_c|)` instead of `mean(|r_c|)`. Producer: extend the existing
`reward_component_ema_kernel` and `popart_component_ema_kernel` to also
emit variance via Welford's online algorithm (single-pass, no extra
device memory).

**Why slot 360 (amendment 2026-05-04 post-B1b smoke):** B1b's structural
refactor revealed that pre-SP11's `reward_components_per_sample[+0]`
(which feeds slot 63 = `REWARD_POPART_EMA_INDEX`) was being overloaded
to mean both "PopArt's normalization input" (= total reward magnitude
EMA) AND "popart-component-specific magnitude". Pre-SP11 these
collapsed to the same value because reward composition was inline
accumulation with popart as the dominant component.

Post-B1b with explicit per-component decomposition, the SP11 mag-ratio
canary reading slot 63 saw the inflated TOTAL-reward magnitude as
"popart's signal", and the controller redistributed weight toward
popart (w_pop ≈ 2.0) based on a contaminated ratio. Symptom: B1b smoke
showed sharpe ≈ 3 vs B1a baseline ≈ 30, traced to the controller
amplifying a wrongly-attributed signal while attenuating the actually-
firing components.

Resolution: allocate slot 360 for popart-component-specific magnitude
EMA. Slot 63 stays as PopArt's input (= total reward magnitude,
pre-SP11 invariant preserved). The SP11 mag-ratio canary reads slot
360 (popart-specific) + slots 64..68 (existing cf/trail/micro/opp_cost/
bonus) to compute the 6-component ratios. No change to PopArt's
existing readers; one new producer kernel + one mag-ratio canary
signature change.

## 5. Producer kernels (4 total)

1. `reward_subsystem_controller_kernel.cu` (the main controller, ~200 LOC)
2. `val_sharpe_delta_compute_kernel.cu` (canary — reads val sharpe history,
   writes delta_ema + delta_var_ema (true variance), ~80 LOC)
3. `saboteur_engagement_compute_kernel.cu` (canary — per-bar
   |reward_with_saboteur − reward_without_saboteur| > EPS_ENGAGEMENT
   reduced via block tree-reduce, EMA'd; see §3.3.1 ~100 LOC)
4. `reward_component_grad_ratio_compute_kernel.cu` (canary — reads
   per-component grad EMA from existing slots, normalizes to ratios, ~60 LOC)

Plus modifications to existing kernels:
- `mse_loss_kernel.cu`, `c51_loss_kernel.cu` — read cf_weight from ISV
- Reward shaping kernels touched during audit — read all 6 component weights
- New: curiosity bonus integration in the reward path
- `gpu_experience_collector.rs` — multiply saboteur_perturbation_scale by ISV mult

## 6. Tests

- Unit test: synthetic z=0 → curiosity at midpoint; z<0 → curiosity rises toward curiosity_bound; z>0 → curiosity drops to floor (0.2 × curiosity_bound), NOT zero — assert `curiosity_pressure > 0` even at z=+10
- Unit test: weight floor ≥ WEIGHT_HARD_FLOOR always
- Unit test: saboteur_intensity ∈ [SABOTEUR_MIN, SABOTEUR_MAX]
- Contract test: 20 new slots have FoldReset entries + dispatch arms
- Layout fingerprint: ISV_TOTAL_DIM = 360
- L40S smoke: 5-epoch run shows weights drifting from initial sentinels,
  curiosity oscillating with stagnation signal, saboteur_intensity reactive

## 7. Design notes and open questions

- **Curiosity is a feature, not a hazard**: The current pathology is that
  the model converges to a fixed point at ep1 and then declines. Curiosity-
  driven exploration is the corrective mechanism — we *want* the model to
  keep exploring before settling. The `CURIOSITY_BOUND_INDEX = 0.3 × PnL
  EMA` is not a defensive cap against curiosity; it's a signal-relative
  scale that keeps the bonus magnitude commensurate with the PnL signal so
  it doesn't drown out the primary objective. The permanent floor pattern
  (per `pearl_blend_formulas_must_have_permanent_floor`) ensures curiosity
  never fully zeros out — exploration pressure is maintained even after
  convergence, which is exactly what addresses the ep1-peak overfitting.
  In other words: more curiosity when stagnant (stagnation breaker), some
  curiosity always (overfitting prevention).
- **Component starvation guard**: Per-component weight reweighting could in
  principle starve a contributor. Mitigated by adaptive floor
  `≥ 0.5 × min_grad_ratio` and post-floor mean-normalization to
  `mean(weights) = 1` (§3.4.3), ensuring every component keeps a
  non-trivial budget without collapsing reward magnitude (mean=1 preserves
  pre-SP11 absolute scale on average; `Σ=1` would cause 6× collapse on
  mutually-exclusive components).
- **Saboteur intensity adapts both directions**: When val sharpe is
  improving, saboteur intensity rises to keep the policy under realistic
  adversarial load (you don't want a model that only works when execution
  is perfect). When stagnating, intensity drops so the policy gets a
  cleaner gradient. The [0.5, 2.0] hard bound and engagement-rate self-
  correction (per `pearl_engagement_rate_self_correction`) keep this
  bounded.
- **First-fold cold-start**: With only 1-2 val emits available, the
  Z-score is noisy. Mitigated by Pearl A sentinel-bootstrap (first
  observation replaces sentinel directly) and Pearl D Wiener-α (smooth as
  data accumulates).
- **Open question**: Should the saboteur intensity also be ISV-driven on a
  PER-CURRICULUM-DIMENSION basis (separate spread / fill / slippage
  multipliers)? Current design uses a single scalar mult. Defer to SP11.1
  if needed after empirical results.

## 8. Implementation phases — full straight-up implementation

Two layers, matching SP4/SP5/SP7. Layer A is additive infrastructure
(no behavior change) split per producer for independent revertability;
Layer B is the atomic consumer migration that flips the trainer onto
the new controller. Smoke runs only at the end of each layer for
validation — there is no gating preflight that pauses or defers work.

### Layer A — additive infrastructure (3 commits, no behavior change)

**A0 — slot allocation + state reset registry (~150 LOC)**
- Allocate 20 new ISV slots (340..360) in `sp5_isv_slots.rs`
- Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`
- 20 FoldReset entries + 20 dispatch arms in state-reset registry
- Novelty hash table reset entry (1M-slot device buffer, fold-cleared)
- Layout fingerprint bumped
- Contract test passes (slot count + reset arms match)

**A1 — three canary producer kernels (~250 LOC)**
- `val_sharpe_delta_compute_kernel.cu` (Δ + variance EMA, two-pass)
- `saboteur_engagement_compute_kernel.cu` (per-bar engagement, Pearl C)
- `reward_component_grad_ratio_compute_kernel.cu` (existing
  per-component grad EMAs → normalized ratios)
- All Pearl-A bootstrapped, Pearl-D smoothed
- `build.rs` cubin registration; Rust launchers in `gpu_dqn_trainer.rs`
- Wire launches into `training_loop.rs`

**A2 — controller producer kernel (~250 LOC)**
- `reward_subsystem_controller_kernel.cu` reads 5 canary slots,
  writes 10 outputs to scratch, chained `apply_pearls_ad_kernel`
  smooths scratch into ISV
- Novelty hash table device buffer (1M slots × f32) + 42×16 SimHash
  projection matrix (deterministic seed, init at trainer construct)
- Build.rs entry, launcher, training_loop wire-up
- HEALTH_DIAG `sp11_reward` line emits 10 outputs + `improvement_z`

### Layer B — split into B0/B1a/B1b/B1c (amendment 2026-05-04 post-A2 audit)

`feedback_no_partial_refactor` requires every consumer of a CHANGED
CONTRACT to migrate together. Layer B is decomposed into four commits
because the original "single atomic Layer B" mixed three independent
contract changes (controller weight semantics, saboteur multiplier,
replay-time curiosity) — splitting them lets each contract migrate
atomically without forcing a 1500-LOC blast in one commit, and lets
B0 (controller-renorm change) precede B1b (which depends on the new
mean=1 semantics).

**B0 — A2 controller renormalization amendment (~30 LOC)**
Per §3.4.3, change the A2-landed controller from `Σweights = 1` to
`mean(weights) = 1` (i.e., `Σ = N = 6`). Modify
`reward_subsystem_controller_kernel.cu` renormalization step to multiply
by `N` after the divide. Add `MAX_WEIGHT = 3.0` Invariant-1 cap. Update
the 3 A2 unit tests' assertions (mean-around-1 instead of sum-to-1).
Smoke verifies HEALTH_DIAG `sp11_reward` shows weights ≈ 1.0 each on
uniform-grad-ratio inputs.

**B1a — small/safe migrations (~150 LOC)**
- Saboteur GPU multiplication: `saboteur_generate_params` kernel reads
  `isv[SABOTEUR_INTENSITY_MULT_INDEX]` on-device, computes
  `effective_scale = base × mult`. `gpu_experience_collector.rs:3328`
  passes ISV pointer + slot index to the kernel; no host-side multiply.
- SimHash kernel `state_stride` parameter added (states_buf is
  STATE_DIM_PADDED=128, but kernel needs to read 42 contiguous floats
  per sample). Kernel signature gains `int state_stride`; call sites
  pass `STATE_DIM_PADDED`.

**B1b — structural reward-composition refactor (~500 LOC, atomic)**
- `experience_env_step` reward decomposition per §3.5.3 (8+ inline sites
  → per-component locals → weighted Σ).
- §3.5.4 r_trail extraction: segregate trail-fire bars from voluntary-
  exit bars in `segment_complete` P&L credit.
- §3.4.4 universal post-composition modifiers: drawdown/inventory/
  churn/conviction-scale/cf-flip applied AFTER `Σ w_i × r_i`,
  unweighted.
- `mse_loss_kernel.cu:318` and `c51_loss_kernel.cu:789` `cf_weight =
  0.3f` constants UNTOUCHED per §3.5 amendment (they're structural
  Q-blend, not reward weights).
- Stale doc-string sweep across modified sites.
- Smoke verifies HEALTH_DIAG `sp11_reward` weights drift across epochs
  AND popart EMA tracks ≈ pre-SP11 magnitude (mean=1 normalization
  preserves scale).

**B1c — replay-time curiosity (post-Layer-C-audit, ~150 LOC)**
- Pre-requisite: Layer C audit per §3.5.5 confirms `trainer.rewards_buf`
  is the single read site for all replay-reward consumers.
- Trainer launches lookup → apply_curiosity_bonus → update sequence
  immediately after `upload_batch_gpu` returns base rewards.
- New kernel `apply_curiosity_bonus_kernel.cu` (in-place
  `r += curiosity_p × novelty(state, action)` on `trainer.rewards_buf`).
- Pre-existing `gather_f32_scalar` in `ml-dqn` left untouched (semantic
  equivalence preserved without cross-crate refactor).

### Layer C — validation + close-out

**C1** — L40S 5-epoch smoke (`multi_fold_convergence`) on Layer B HEAD
**C2** — full T10 (3-seed × 3-fold × 50-epoch) on Layer B HEAD
**C3** — audit doc: Fix 39 entry with architecture summary + canary
        evolution evidence + Layer B before/after metrics
**C4** — `pearl_reward_as_controlled_subsystem.md` authored
**C5** — MEMORY.md index entry

**Total estimated:** Layer A ≈ 650 LOC + Layer B ≈ 750 LOC + Layer C
≈ 150 LOC = ~1550 LOC across ~12 files. Layer A: ~1.5 hours. Layer B:
~1 hour. Layer C: ~30 min. Plus validation wall-clock (smoke ≈ 25 min,
T10 ≈ 4 hr).

## 9. Validation criteria

### 9.1 Smoke (5-epoch L40S `multi_fold_convergence`)

Hard-pass criteria (all must hold):
- HEALTH_DIAG `sp11_reward` line emits every epoch with all weights in
  [adaptive_floor, MAX_WEIGHT=3.0] and `mean(weights) ≈ 1.0 ± 1e-3`
  (i.e., `Σ ≈ 6.0 ± 6e-3` with N=6 components, per §3.4.3)
- `improvement_z` is non-zero by ep2 (canary populates after Pearl A
  bootstrap on ep1's first delta)
- `curiosity_pressure ≥ 0.2 × curiosity_bound` always (permanent floor)
  and `≤ curiosity_bound` always
- `saboteur_intensity ∈ [0.5, 2.0]` always (post-clamp)
- No regression in Fix 33-38 metrics: IQN parallel-branch warnings = 0,
  dir_entropy > 0.5, val_active_frac > 0.20

### 9.2 T10 (3-seed × 3-fold × 50-epoch L40S)

Aggregation: 9 trajectories. Per-trajectory peak-epoch is the epoch
maximizing val sharpe within that trajectory.

Hard-pass criteria (all must hold):
- **Median peak-epoch across 9 trajectories ≥ 10**
  (Fix 33-38 baseline: median peak-epoch = 1, where xkjkb seed-0 fold-0
  peaked at ep1.) This is the primary signal that the model continues
  learning past ep1.
- **Median val sharpe at ep20 > 0.95 × median val sharpe at peak-epoch**
  (the controller may shift the peak forward; a 5% drop from peak by ep20
  is acceptable noise. A larger drop indicates regression.)
- **Mean val sharpe across all 9 trajectories at ep20 ≥ Fix 33-38 baseline
  ep1 mean** (sanity check — SP11 cannot make ep20 *worse* than the
  baseline's best epoch.)

Soft criteria (informative, not gating):
- Component weight evolution visible across training (different
  components dominate at different stages — std(weight[c]) > floor over
  full T10)
- Q-value growth correlated with val sharpe (Pearson(train Q, val sharpe)
  > 0.3 within each trajectory; baseline showed Pearson = -0.7 at the
  ep0-14 window — the decoupled-overfitting signature)
- Per-fold pattern: fold 0 may peak earliest, fold 2 latest (cumulative
  novelty hash effects), but no fold collapses (peak-epoch ≥ 5 in every
  fold of every seed)

### 9.3 If T10 fails

Fix-forward, not roll-back. Failure modes and responses:
- Median peak-epoch < 10 but z-canary populated: controller has signal
  but isn't acting strongly enough — extend ISV outputs to additional
  shaping multipliers found during Layer B audit
- z-canary stuck at 0: variance EMA is mis-tracking; review Pearl A
  bootstrap on `VAL_SHARPE_VAR_EMA`
- Component weights collapse to floor: mean-normalization or floor
  computation bug; review §3.4 implementation
- Curiosity dominates loss (`r_total / r_base > 1.0` median):
  CURIOSITY_BOUND_FRACTION too high; this is the only Invariant-1
  fraction we'd revisit and still not because we hardcoded a regime,
  but because the chosen rate-limiter was too permissive

## 10. New pearl: `pearl_reward_as_controlled_subsystem`

Authored alongside this SP. Captures the meta-principle:

> Every degree of freedom in the reward path is an output of a unified
> controller whose inputs are improvement-rate canaries. Reward components,
> shaping multipliers, exploration bonuses, and curriculum knobs are all
> ISV-driven from a Z-score-relative improvement signal that never depends
> on hardcoded thresholds. Permanent floors prevent any component from
> being silently zeroed out; OR'd canaries prevent any single signal-failure
> from stalling the controller. This subsumes the controller-anchor pearl
> applied at the reward layer.
