spec(sp11): reward as controlled subsystem — fully ISV-driven

Brainstorm spec for SP11. Resolves the policy-stagnation pathology surfaced in T10 train-multi-seed-xkjkb seed-0 ep0-14: model finds a stable fixed point at ep1 (peak val sharpe 80.61), then OVERFITS to it across remaining epochs (decline 80.61 → 70.58). Q-values grow but val performance declines because reward function has no improvement pressure. Architecture (every input ISV-driven): - Z-score-driven adaptation (no hardcoded "improving" threshold): improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS) - 10 ISV outputs: 6 component weights + curiosity_pressure + saboteur_intensity_mult + adaptive weight_floor + curiosity_bound - 5 ISV canaries: val_sharpe_delta + val_sharpe_std (Z-score noise estimate) + 6 per-component grad ratios + saboteur engagement + PnL magnitude EMA (signal-relative curiosity bound) - 4 new producer kernels (controller + 3 canary computers) - Audit + migrate hardcoded cf_weight=0.3 in mse_loss_kernel.cu:318 and c51_loss_kernel.cu:789, plus other shaping multipliers - NEW reward dimension: curiosity bonus, bounded by PnL magnitude Per pearl_controller_anchors_isv_driven: every threshold replaced with sigmoid(z) — no constants encode "what counts as improving". Per pearl_blend_formulas_must_have_permanent_floor: every weight has adaptive floor preventing zero-out. Per pearl_engagement_rate_self_ correction: saboteur intensity self-corrects via engagement rate canary. Per pearl_cold_start_exit_signal_or: improvement signal OR'd from multiple canaries so single-signal-failure doesn't stall controller. New pearl authored alongside spec: pearl_reward_as_controlled_subsystem — meta-principle that every reward path degree of freedom is a unified controller output. Subsumes controller-anchor pearl at the reward layer. Scope: 20 ISV slots, 4 producer kernels, audit + migration of hardcoded reward shaping constants, 1 atomic commit. ~1300-1700 LOC; ~2.5-3 hours subagent work. Success metric: val_sharpe[ep20] > val_sharpe[ep1] (Fix 33-38 baseline peaked at ep1; SP11 should shift peak later as model continues learning).
2026-05-04 00:04:12 +02:00
parent e580c1388c
commit a72a23e6b1
1 changed files with 325 additions and 0 deletions
--- a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md
+++ b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md
@@ -0,0 +1,325 @@
+# SP11 — Reward as Controlled Subsystem (fully ISV-driven)
+
+**Date:** 2026-05-04
+**Status:** Draft (user review pending)
+**Authors:** session-1777830566225
+**Related pearls:**
+- `pearl_controller_anchors_isv_driven.md` (every threshold ISV-driven)
+- `pearl_cold_start_exit_signal_or.md` (OR'd canary signals)
+- `pearl_blend_formulas_must_have_permanent_floor.md` (floors prevent zero-out)
+- `pearl_engagement_rate_self_correction.md` (saboteur engagement self-corrects)
+- `pearl_wiener_optimal_adaptive_alpha.md` (Pearl D — α from variance ratio)
+- `pearl_first_observation_bootstrap.md` (Pearl A sentinel-bootstrap)
+
+## 1. Overview
+
+The Fix 33-38 chain made every CONTROLLER signal-driven (CQL budget, MAX_BUDGET,
+Kelly cap, Thompson temperature). The reward subsystem is the last
+regime-encoded layer in the training stack. SP11 promotes every reward
+component weight, shaping multiplier, exploration bonus, and curriculum knob
+to ISV-driven slots. **Every input is itself ISV-driven** — no hardcoded
+threshold, no constant floor that isn't an Invariant-1 numerical anchor.
+
+The new pearl this authors (`pearl_reward_as_controlled_subsystem`) declares
+the principle: every degree of freedom in the reward path is an output of a
+unified controller whose inputs are improvement-rate canaries.
+
+## 2. Pathology
+
+T10 train-multi-seed-xkjkb seed-0 ep0-14 evidence (commit `920a2d021`,
+all SP10 + earlier fixes applied):
+
+```
+ep0:  val sharpe=75.18 trade_count=134327 active_frac=0.502 dir_entropy=1.041
+ep1:  val sharpe=80.61 trade_count=133997 active_frac=0.500 dir_entropy=1.039  ← peak
+ep2:  val sharpe=77.64 trade_count=131911 active_frac=0.481 dir_entropy=1.025
+...
+ep10: val sharpe=73.10 trade_count=120617 active_frac=0.403 dir_entropy=0.953
+ep14: val sharpe=70.58 trade_count=117236 active_frac=0.384 dir_entropy=0.932
+
+train: Q-value 0.04 → 0.30 (growing), train_loss stable 3.7-4.1
+```
+
+**Diagnosis:** the model finds a stable fixed point at ep1 then OVERFITS to
+it. Q-values grow (TD targets propagate) but val performance declines because
+the policy has nothing pushing it to be _better_, only to be _consistent_.
+The reward function is purely "how good was this bar" with no improvement
+pressure, no curriculum gradient, no curiosity.
+
+The reward components (`popart`, `cf`, `trail_r`, `micro`, `opp_cost`,
+`bonus`) all have hardcoded weights (`cf_weight = 0.3f` at
+`mse_loss_kernel.cu:318` and `c51_loss_kernel.cu:789`; saboteur intensity
+decays on a hardcoded 0.995/epoch schedule in `gpu_experience_collector.rs`).
+None adapt to whether training is improving, stagnating, or regressing.
+
+## 3. Architecture (refined: every input ISV-driven)
+
+### 3.1 Z-score-driven adaptation (no hardcoded thresholds)
+
+The "improving / stagnant / regressing" classification uses a Z-score, not a
+hardcoded threshold:
+
+```
+improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS_DIV)
+```
+
+Where:
+- `val_sharpe_delta_ema` = EMA of (val_sharpe[t] − val_sharpe[t-1])
+- `val_sharpe_std_ema` = EMA of |val_sharpe_delta| (running noise estimate)
+- Both Pearl D Wiener-α-smoothed; both have Pearl A sentinel-bootstrap
+
+`improvement_z` ∈ ℝ; the controller's outputs are continuous functions of
+`z`, not threshold-gated:
+- `z > 0`: improving — exploit current, raise curriculum, reweight toward
+  contributing components
+- `z ≈ 0`: stagnant — raise curiosity, raise saboteur intensity, diversify
+  weights
+- `z < 0`: regressing — max curiosity, lower saboteur (relax curriculum),
+  return weights to floor mean
+
+### 3.2 Outputs (10 ISV slots)
+
+| Slot | Purpose | Floor |
+|---|---|---|
+| `REWARD_POPART_WEIGHT_INDEX` | PnL-magnitude reward weight | signal-driven |
+| `REWARD_CF_WEIGHT_INDEX` | Counterfactual reward weight | signal-driven |
+| `REWARD_TRAIL_WEIGHT_INDEX` | Trailing-stop reward weight | signal-driven |
+| `REWARD_MICRO_WEIGHT_INDEX` | OFI micro-structure reward weight | signal-driven |
+| `REWARD_OPP_COST_WEIGHT_INDEX` | Flat opportunity cost weight | signal-driven |
+| `REWARD_BONUS_WEIGHT_INDEX` | Trade-attempt/timing bonus weight | signal-driven |
+| `CURIOSITY_PRESSURE_INDEX` | NEW exploration-bonus magnitude | bounded by `pnl_reward_magnitude_ema` |
+| `SABOTEUR_INTENSITY_MULT_INDEX` | Curriculum modulator on saboteur scale | Invariant-1 [0.5, 2.0] |
+| `REWARD_WEIGHT_FLOOR_INDEX` | Per-component minimum weight | signal-driven from grad-history |
+| `CURIOSITY_BOUND_INDEX` | Max curiosity bonus magnitude | signal-driven from PnL EMA |
+
+### 3.3 Canaries (5 ISV slots)
+
+| Slot | Source | Pearl pattern |
+|---|---|---|
+| `VAL_SHARPE_DELTA_EMA_INDEX` | `Δsharpe[t] = sharpe[t]−sharpe[t-1]`, Pearl D smoothed | A+D |
+| `VAL_SHARPE_STD_EMA_INDEX` | EMA of `|Δsharpe|` (noise estimate) | A+D |
+| `REWARD_COMPONENT_GRAD_RATIO_BASE` | 6 slots, per-component grad-EMA / total-grad-EMA | A+D |
+| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Fraction of episodes where saboteur params shifted Q by > EPS | Pearl C engagement counter |
+| `PNL_REWARD_MAGNITUDE_EMA_INDEX` | `|popart_reward|` EMA — bounds curiosity proportionally | A+D |
+
+### 3.4 Producer kernel: `reward_subsystem_controller`
+
+Single block, 10 threads (one per output). Reads all 5 canaries, computes:
+
+```cuda
+const float EPS_DIV = 1e-6f;
+const float SABOTEUR_MIN = 0.5f;        // Invariant-1 rate limiter
+const float SABOTEUR_MAX = 2.0f;        // Invariant-1 rate limiter
+const float WEIGHT_HARD_FLOOR = 0.01f;  // Invariant-1 numerical floor (prevents zero)
+
+// Z-score (signal-relative; no hardcoded threshold)
+float dz = isv[VAL_SHARPE_DELTA_EMA] / fmaxf(isv[VAL_SHARPE_STD_EMA], EPS_DIV);
+
+// Sigmoid maps z continuously to [0, 1]; no thresholds
+float improving = 1.0f / (1.0f + expf(-dz));    // 0=regressing, 0.5=stagnant, 1=improving
+float stagnant_or_worse = 1.0f - improving;
+
+// Per-component weights: blend toward (a) high-grad-ratio winners when
+// improving, (b) under-used components when stagnant, (c) floor when
+// regressing. Floor itself signal-driven.
+float floor = isv[REWARD_WEIGHT_FLOOR_INDEX];   // adaptive (see below)
+for (int c = 0; c < 6; c++) {
+    float ratio = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + c];
+    float winner_weight = ratio;                                  // exploit
+    float diversifier_weight = (1.0f - ratio) / 5.0f;              // explore
+    float blend = improving * winner_weight + stagnant_or_worse * diversifier_weight;
+    float weight = fmaxf(floor, fminf(1.0f, blend));
+    scratch[scratch_weight_base + c] = weight;
+}
+
+// Curiosity bound = fraction of recent PnL magnitude (signal-relative)
+float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f;     // 0.3 is Invariant-1 fraction
+scratch[scratch_curiosity_bound] = curiosity_bound;
+
+// Curiosity pressure: scales [0, curiosity_bound] by (1 - improving)
+float curiosity_pressure = stagnant_or_worse * curiosity_bound;
+scratch[scratch_curiosity_pressure] = curiosity_pressure;
+
+// Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX]
+// adjusted by engagement rate (per pearl_engagement_rate_self_correction).
+// If saboteur is rarely shifting Q (engagement → 0), back off.
+float engagement = isv[SABOTEUR_ENGAGEMENT_RATE_INDEX];
+float intensity_z_term = SABOTEUR_MIN + (SABOTEUR_MAX - SABOTEUR_MIN) * improving;
+float saboteur_intensity = intensity_z_term * fmaxf(engagement, 0.1f);  // 0.1 is engagement floor
+scratch[scratch_saboteur_intensity] = saboteur_intensity;
+
+// Per-component weight floor: signal-driven from worst-suppressed component's
+// grad-EMA. Idea: if any component has been consistently below mean grad,
+// its floor rises so it gets a chance to contribute.
+float min_grad_ratio = ratios[0];
+for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]);
+float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio);  // 0.5 = Invariant-1 fraction
+scratch[scratch_weight_floor] = adaptive_floor;
+```
+
+The constants `0.3f` (curiosity fraction), `0.5f` (floor fraction), and
+`0.1f` (engagement floor) are Invariant-1 fractions: they bound rates, not
+regimes, and remain valid across all regimes. The thresholds themselves
+(what counts as "improving", "stagnant", "regressing") are entirely
+encoded in the Z-score and sigmoid — no hardcoded threshold survives.
+
+### 3.5 Consumer migrations
+
+**Reward shaping kernels** (consume per-component weights from ISV):
+- `mse_loss_kernel.cu:318` — replace `cf_weight = 0.3f` with
+  `isv[REWARD_CF_WEIGHT_INDEX]`
+- `c51_loss_kernel.cu:789` — same migration
+- Other reward shaping multipliers identified during implementation audit
+
+**Curiosity bonus** (NEW reward dimension):
+- New term in reward computation: `r_total += isv[CURIOSITY_PRESSURE_INDEX] *
+  novelty_signal(state, action)`
+- `novelty_signal` is the simplest workable form: state-action visit count
+  EMA → 1 / sqrt(count) (RND-style optional later)
+- Bounded by `CURIOSITY_BOUND_INDEX` in the producer kernel
+
+**Saboteur intensity** (modulator on existing saboteur scale):
+- `gpu_experience_collector.rs:3328` — `let ps =
+  self.saboteur_perturbation_scale * isv[SABOTEUR_INTENSITY_MULT_INDEX]` (was
+  raw `saboteur_perturbation_scale`)
+- The existing 0.995/epoch decay schedule remains as the BASE; the ISV
+  multiplier is the adaptive _curriculum modulator_ on top
+
+### 3.6 Reset semantics
+
+All 15 new slots: FoldReset (sentinel 0, Pearl A bootstraps from first
+observation). Cross-fold persistence is wrong here because the
+"improvement signal" should reset at each fold's cold-start.
+
+**Exception:** `REWARD_COMPONENT_GRAD_RATIO_BASE` slots — these accumulate
+across the fold and reset to sentinel at fold boundary. The other slots
+(`val_sharpe_delta`, `saboteur_engagement_rate`) need at least 2 epochs
+within a fold to compute a delta; they're signal-bootstrapped so the first
+epoch of each fold uses the floor (no signal yet).
+
+## 4. ISV slot allocation
+
+After SP10's last slot at 339:
+
+| Slot range | Purpose | Count |
+|---|---|---|
+| `[340..346)` | 6 component weights | 6 |
+| `346` | `CURIOSITY_PRESSURE_INDEX` | 1 |
+| `347` | `SABOTEUR_INTENSITY_MULT_INDEX` | 1 |
+| `348` | `REWARD_WEIGHT_FLOOR_INDEX` (adaptive) | 1 |
+| `349` | `CURIOSITY_BOUND_INDEX` (signal-relative) | 1 |
+| `350` | `VAL_SHARPE_DELTA_EMA_INDEX` (canary) | 1 |
+| `351` | `VAL_SHARPE_STD_EMA_INDEX` (canary) | 1 |
+| `[352..358)` | `REWARD_COMPONENT_GRAD_RATIO_BASE` (6 canaries) | 6 |
+| `358` | `SABOTEUR_ENGAGEMENT_RATE_INDEX` (canary) | 1 |
+| `359` | `PNL_REWARD_MAGNITUDE_EMA_INDEX` (canary) | 1 |
+
+Total: **20 new slots**. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`.
+
+## 5. Producer kernels (4 total)
+
+1. `reward_subsystem_controller_kernel.cu` (the main controller, ~200 LOC)
+2. `val_sharpe_delta_compute_kernel.cu` (canary — reads val sharpe history,
+   writes Δ + |Δ| EMAs, ~80 LOC)
+3. `saboteur_engagement_compute_kernel.cu` (canary — reads per-episode Q
+   shift after saboteur params change, writes engagement rate, ~100 LOC)
+4. `reward_component_grad_ratio_compute_kernel.cu` (canary — reads
+   per-component grad EMA from existing slots, normalizes to ratios, ~60 LOC)
+
+Plus modifications to existing kernels:
+- `mse_loss_kernel.cu`, `c51_loss_kernel.cu` — read cf_weight from ISV
+- Reward shaping kernels touched during audit — read all 6 component weights
+- New: curiosity bonus integration in the reward path
+- `gpu_experience_collector.rs` — multiply saboteur_perturbation_scale by ISV mult
+
+## 6. Tests
+
+- Unit test: synthetic z=0 → curiosity at midpoint; z>0 → curiosity drops; z<0 → curiosity max
+- Unit test: weight floor ≥ WEIGHT_HARD_FLOOR always
+- Unit test: saboteur_intensity ∈ [SABOTEUR_MIN, SABOTEUR_MAX]
+- Contract test: 20 new slots have FoldReset entries + dispatch arms
+- Layout fingerprint: ISV_TOTAL_DIM = 360
+- L40S smoke: 5-epoch run shows weights drifting from initial sentinels,
+  curiosity oscillating with stagnation signal, saboteur_intensity reactive
+
+## 7. Risks and open questions
+
+- **Risk 1**: Curiosity bonus could destabilize a working policy. Mitigated
+  by `CURIOSITY_BOUND_INDEX` = 0.3 × PnL EMA — curiosity is always at most
+  30% of PnL signal magnitude. Permanent floor pattern means it never
+  zeros out either, so we always have some exploration.
+- **Risk 2**: Per-component weight reweighting could starve a component.
+  Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the
+  least-contributing component keeps a non-trivial budget.
+- **Risk 3**: Saboteur intensity rising on improvement could over-stress the
+  policy. Mitigated by [0.5, 2.0] hard bound and engagement-rate self-
+  correction (per `pearl_engagement_rate_self_correction`).
+- **Risk 4**: First-fold cold-start: with only 1-2 val emits available, the
+  Z-score is noisy. Mitigated by Pearl A sentinel-bootstrap (first
+  observation replaces sentinel directly) and Pearl D Wiener-α (smooth as
+  data accumulates).
+- **Open question**: Should the saboteur intensity also be ISV-driven on a
+  PER-CURRICULUM-DIMENSION basis (separate spread / fill / slippage
+  multipliers)? Current design uses a single scalar mult. Defer to SP11.1
+  if needed after empirical results.
+
+## 8. Implementation phases (atomic per `feedback_no_partial_refactor`)
+
+Single commit:
+
+1. Allocate 20 new ISV slots in `sp5_isv_slots.rs` (range 340..360)
+2. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`, layout fingerprint
+3. Add 4 new producer kernels (controller + 3 canaries)
+4. Register kernels in `build.rs`
+5. Rust launchers in `gpu_dqn_trainer.rs` for all 4 new kernels
+6. Wire launches into `training_loop.rs` at appropriate points
+7. **Audit + migrate ALL existing reward weight constants:**
+   - `mse_loss_kernel.cu:318` (cf_weight = 0.3 → ISV)
+   - `c51_loss_kernel.cu:789` (cf_weight = 0.3 → ISV)
+   - Other shaping multipliers found during audit
+8. Add curiosity bonus term to reward path (NEW dimension)
+9. Modify `gpu_experience_collector.rs:3328` for saboteur_intensity multiplier
+10. State reset registry: 20 new FoldReset entries + dispatch arms
+11. Contract test passes
+12. New pearl: `pearl_reward_as_controlled_subsystem.md` authored
+13. MEMORY.md index entry added
+14. Audit doc: Fix 39 entry with full architecture summary
+15. HEALTH_DIAG line for `sp11_reward` showing all 10 outputs + improvement_z
+
+Total estimated: ~1300-1700 LOC across ~12 files. ~2.5-3 hours subagent work.
+
+## 9. Validation criteria
+
+After SP11 lands:
+
+**Smoke (5-epoch L40S):**
+- HEALTH_DIAG `sp11_reward` line emits with all weights in [floor, 1.0]
+- `improvement_z` evolves across epochs (not stuck at 0)
+- `curiosity_pressure` oscillates with stagnation
+- No regression in Fix 33-38 metrics (IQN warnings still 0, dir_entropy
+  still > 0.5, val_active_frac still > 0.20)
+
+**T10 (50-epoch full validation):**
+- val sharpe trajectory shows continued IMPROVEMENT (not just stagnation
+  or decline) at least into ep20
+- Q-value continues to grow with val sharpe (not decoupled like ep0-14
+  baseline)
+- Component weight evolution visible in HEALTH_DIAG (different components
+  win at different training stages)
+
+**Specific success metric:** `val_sharpe[ep20] > val_sharpe[ep1]`. The Fix
+33-38 baseline showed `ep1 > ep14` (peak at ep1, declining). SP11
+success = peak shifts to ep20+ or beyond.
+
+## 10. New pearl: `pearl_reward_as_controlled_subsystem`
+
+Authored alongside this SP. Captures the meta-principle:
+
+> Every degree of freedom in the reward path is an output of a unified
+> controller whose inputs are improvement-rate canaries. Reward components,
+> shaping multipliers, exploration bonuses, and curriculum knobs are all
+> ISV-driven from a Z-score-relative improvement signal that never depends
+> on hardcoded thresholds. Permanent floors prevent any component from
+> being silently zeroed out; OR'd canaries prevent any single signal-failure
+> from stalling the controller. This subsumes the controller-anchor pearl
+> applied at the reward layer.