From a72a23e6b1f1ec14d56ccb57c75fbbc9360dfec9 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 4 May 2026 00:04:12 +0200 Subject: [PATCH] =?UTF-8?q?spec(sp11):=20reward=20as=20controlled=20subsys?= =?UTF-8?q?tem=20=E2=80=94=20fully=20ISV-driven?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brainstorm spec for SP11. Resolves the policy-stagnation pathology surfaced in T10 train-multi-seed-xkjkb seed-0 ep0-14: model finds a stable fixed point at ep1 (peak val sharpe 80.61), then OVERFITS to it across remaining epochs (decline 80.61 → 70.58). Q-values grow but val performance declines because reward function has no improvement pressure. Architecture (every input ISV-driven): - Z-score-driven adaptation (no hardcoded "improving" threshold): improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS) - 10 ISV outputs: 6 component weights + curiosity_pressure + saboteur_intensity_mult + adaptive weight_floor + curiosity_bound - 5 ISV canaries: val_sharpe_delta + val_sharpe_std (Z-score noise estimate) + 6 per-component grad ratios + saboteur engagement + PnL magnitude EMA (signal-relative curiosity bound) - 4 new producer kernels (controller + 3 canary computers) - Audit + migrate hardcoded cf_weight=0.3 in mse_loss_kernel.cu:318 and c51_loss_kernel.cu:789, plus other shaping multipliers - NEW reward dimension: curiosity bonus, bounded by PnL magnitude Per pearl_controller_anchors_isv_driven: every threshold replaced with sigmoid(z) — no constants encode "what counts as improving". Per pearl_blend_formulas_must_have_permanent_floor: every weight has adaptive floor preventing zero-out. Per pearl_engagement_rate_self_ correction: saboteur intensity self-corrects via engagement rate canary. Per pearl_cold_start_exit_signal_or: improvement signal OR'd from multiple canaries so single-signal-failure doesn't stall controller. New pearl authored alongside spec: pearl_reward_as_controlled_subsystem — meta-principle that every reward path degree of freedom is a unified controller output. Subsumes controller-anchor pearl at the reward layer. Scope: 20 ISV slots, 4 producer kernels, audit + migration of hardcoded reward shaping constants, 1 atomic commit. ~1300-1700 LOC; ~2.5-3 hours subagent work. Success metric: val_sharpe[ep20] > val_sharpe[ep1] (Fix 33-38 baseline peaked at ep1; SP11 should shift peak later as model continues learning). --- ...-04-sp11-reward-as-controlled-subsystem.md | 325 ++++++++++++++++++ 1 file changed, 325 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md diff --git a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md new file mode 100644 index 000000000..48b2fcef2 --- /dev/null +++ b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md @@ -0,0 +1,325 @@ +# SP11 — Reward as Controlled Subsystem (fully ISV-driven) + +**Date:** 2026-05-04 +**Status:** Draft (user review pending) +**Authors:** session-1777830566225 +**Related pearls:** +- `pearl_controller_anchors_isv_driven.md` (every threshold ISV-driven) +- `pearl_cold_start_exit_signal_or.md` (OR'd canary signals) +- `pearl_blend_formulas_must_have_permanent_floor.md` (floors prevent zero-out) +- `pearl_engagement_rate_self_correction.md` (saboteur engagement self-corrects) +- `pearl_wiener_optimal_adaptive_alpha.md` (Pearl D — α from variance ratio) +- `pearl_first_observation_bootstrap.md` (Pearl A sentinel-bootstrap) + +## 1. Overview + +The Fix 33-38 chain made every CONTROLLER signal-driven (CQL budget, MAX_BUDGET, +Kelly cap, Thompson temperature). The reward subsystem is the last +regime-encoded layer in the training stack. SP11 promotes every reward +component weight, shaping multiplier, exploration bonus, and curriculum knob +to ISV-driven slots. **Every input is itself ISV-driven** — no hardcoded +threshold, no constant floor that isn't an Invariant-1 numerical anchor. + +The new pearl this authors (`pearl_reward_as_controlled_subsystem`) declares +the principle: every degree of freedom in the reward path is an output of a +unified controller whose inputs are improvement-rate canaries. + +## 2. Pathology + +T10 train-multi-seed-xkjkb seed-0 ep0-14 evidence (commit `920a2d021`, +all SP10 + earlier fixes applied): + +``` +ep0: val sharpe=75.18 trade_count=134327 active_frac=0.502 dir_entropy=1.041 +ep1: val sharpe=80.61 trade_count=133997 active_frac=0.500 dir_entropy=1.039 ← peak +ep2: val sharpe=77.64 trade_count=131911 active_frac=0.481 dir_entropy=1.025 +... +ep10: val sharpe=73.10 trade_count=120617 active_frac=0.403 dir_entropy=0.953 +ep14: val sharpe=70.58 trade_count=117236 active_frac=0.384 dir_entropy=0.932 + +train: Q-value 0.04 → 0.30 (growing), train_loss stable 3.7-4.1 +``` + +**Diagnosis:** the model finds a stable fixed point at ep1 then OVERFITS to +it. Q-values grow (TD targets propagate) but val performance declines because +the policy has nothing pushing it to be _better_, only to be _consistent_. +The reward function is purely "how good was this bar" with no improvement +pressure, no curriculum gradient, no curiosity. + +The reward components (`popart`, `cf`, `trail_r`, `micro`, `opp_cost`, +`bonus`) all have hardcoded weights (`cf_weight = 0.3f` at +`mse_loss_kernel.cu:318` and `c51_loss_kernel.cu:789`; saboteur intensity +decays on a hardcoded 0.995/epoch schedule in `gpu_experience_collector.rs`). +None adapt to whether training is improving, stagnating, or regressing. + +## 3. Architecture (refined: every input ISV-driven) + +### 3.1 Z-score-driven adaptation (no hardcoded thresholds) + +The "improving / stagnant / regressing" classification uses a Z-score, not a +hardcoded threshold: + +``` +improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS_DIV) +``` + +Where: +- `val_sharpe_delta_ema` = EMA of (val_sharpe[t] − val_sharpe[t-1]) +- `val_sharpe_std_ema` = EMA of |val_sharpe_delta| (running noise estimate) +- Both Pearl D Wiener-α-smoothed; both have Pearl A sentinel-bootstrap + +`improvement_z` ∈ ℝ; the controller's outputs are continuous functions of +`z`, not threshold-gated: +- `z > 0`: improving — exploit current, raise curriculum, reweight toward + contributing components +- `z ≈ 0`: stagnant — raise curiosity, raise saboteur intensity, diversify + weights +- `z < 0`: regressing — max curiosity, lower saboteur (relax curriculum), + return weights to floor mean + +### 3.2 Outputs (10 ISV slots) + +| Slot | Purpose | Floor | +|---|---|---| +| `REWARD_POPART_WEIGHT_INDEX` | PnL-magnitude reward weight | signal-driven | +| `REWARD_CF_WEIGHT_INDEX` | Counterfactual reward weight | signal-driven | +| `REWARD_TRAIL_WEIGHT_INDEX` | Trailing-stop reward weight | signal-driven | +| `REWARD_MICRO_WEIGHT_INDEX` | OFI micro-structure reward weight | signal-driven | +| `REWARD_OPP_COST_WEIGHT_INDEX` | Flat opportunity cost weight | signal-driven | +| `REWARD_BONUS_WEIGHT_INDEX` | Trade-attempt/timing bonus weight | signal-driven | +| `CURIOSITY_PRESSURE_INDEX` | NEW exploration-bonus magnitude | bounded by `pnl_reward_magnitude_ema` | +| `SABOTEUR_INTENSITY_MULT_INDEX` | Curriculum modulator on saboteur scale | Invariant-1 [0.5, 2.0] | +| `REWARD_WEIGHT_FLOOR_INDEX` | Per-component minimum weight | signal-driven from grad-history | +| `CURIOSITY_BOUND_INDEX` | Max curiosity bonus magnitude | signal-driven from PnL EMA | + +### 3.3 Canaries (5 ISV slots) + +| Slot | Source | Pearl pattern | +|---|---|---| +| `VAL_SHARPE_DELTA_EMA_INDEX` | `Δsharpe[t] = sharpe[t]−sharpe[t-1]`, Pearl D smoothed | A+D | +| `VAL_SHARPE_STD_EMA_INDEX` | EMA of `|Δsharpe|` (noise estimate) | A+D | +| `REWARD_COMPONENT_GRAD_RATIO_BASE` | 6 slots, per-component grad-EMA / total-grad-EMA | A+D | +| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Fraction of episodes where saboteur params shifted Q by > EPS | Pearl C engagement counter | +| `PNL_REWARD_MAGNITUDE_EMA_INDEX` | `|popart_reward|` EMA — bounds curiosity proportionally | A+D | + +### 3.4 Producer kernel: `reward_subsystem_controller` + +Single block, 10 threads (one per output). Reads all 5 canaries, computes: + +```cuda +const float EPS_DIV = 1e-6f; +const float SABOTEUR_MIN = 0.5f; // Invariant-1 rate limiter +const float SABOTEUR_MAX = 2.0f; // Invariant-1 rate limiter +const float WEIGHT_HARD_FLOOR = 0.01f; // Invariant-1 numerical floor (prevents zero) + +// Z-score (signal-relative; no hardcoded threshold) +float dz = isv[VAL_SHARPE_DELTA_EMA] / fmaxf(isv[VAL_SHARPE_STD_EMA], EPS_DIV); + +// Sigmoid maps z continuously to [0, 1]; no thresholds +float improving = 1.0f / (1.0f + expf(-dz)); // 0=regressing, 0.5=stagnant, 1=improving +float stagnant_or_worse = 1.0f - improving; + +// Per-component weights: blend toward (a) high-grad-ratio winners when +// improving, (b) under-used components when stagnant, (c) floor when +// regressing. Floor itself signal-driven. +float floor = isv[REWARD_WEIGHT_FLOOR_INDEX]; // adaptive (see below) +for (int c = 0; c < 6; c++) { + float ratio = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + c]; + float winner_weight = ratio; // exploit + float diversifier_weight = (1.0f - ratio) / 5.0f; // explore + float blend = improving * winner_weight + stagnant_or_worse * diversifier_weight; + float weight = fmaxf(floor, fminf(1.0f, blend)); + scratch[scratch_weight_base + c] = weight; +} + +// Curiosity bound = fraction of recent PnL magnitude (signal-relative) +float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f; // 0.3 is Invariant-1 fraction +scratch[scratch_curiosity_bound] = curiosity_bound; + +// Curiosity pressure: scales [0, curiosity_bound] by (1 - improving) +float curiosity_pressure = stagnant_or_worse * curiosity_bound; +scratch[scratch_curiosity_pressure] = curiosity_pressure; + +// Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX] +// adjusted by engagement rate (per pearl_engagement_rate_self_correction). +// If saboteur is rarely shifting Q (engagement → 0), back off. +float engagement = isv[SABOTEUR_ENGAGEMENT_RATE_INDEX]; +float intensity_z_term = SABOTEUR_MIN + (SABOTEUR_MAX - SABOTEUR_MIN) * improving; +float saboteur_intensity = intensity_z_term * fmaxf(engagement, 0.1f); // 0.1 is engagement floor +scratch[scratch_saboteur_intensity] = saboteur_intensity; + +// Per-component weight floor: signal-driven from worst-suppressed component's +// grad-EMA. Idea: if any component has been consistently below mean grad, +// its floor rises so it gets a chance to contribute. +float min_grad_ratio = ratios[0]; +for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]); +float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio); // 0.5 = Invariant-1 fraction +scratch[scratch_weight_floor] = adaptive_floor; +``` + +The constants `0.3f` (curiosity fraction), `0.5f` (floor fraction), and +`0.1f` (engagement floor) are Invariant-1 fractions: they bound rates, not +regimes, and remain valid across all regimes. The thresholds themselves +(what counts as "improving", "stagnant", "regressing") are entirely +encoded in the Z-score and sigmoid — no hardcoded threshold survives. + +### 3.5 Consumer migrations + +**Reward shaping kernels** (consume per-component weights from ISV): +- `mse_loss_kernel.cu:318` — replace `cf_weight = 0.3f` with + `isv[REWARD_CF_WEIGHT_INDEX]` +- `c51_loss_kernel.cu:789` — same migration +- Other reward shaping multipliers identified during implementation audit + +**Curiosity bonus** (NEW reward dimension): +- New term in reward computation: `r_total += isv[CURIOSITY_PRESSURE_INDEX] * + novelty_signal(state, action)` +- `novelty_signal` is the simplest workable form: state-action visit count + EMA → 1 / sqrt(count) (RND-style optional later) +- Bounded by `CURIOSITY_BOUND_INDEX` in the producer kernel + +**Saboteur intensity** (modulator on existing saboteur scale): +- `gpu_experience_collector.rs:3328` — `let ps = + self.saboteur_perturbation_scale * isv[SABOTEUR_INTENSITY_MULT_INDEX]` (was + raw `saboteur_perturbation_scale`) +- The existing 0.995/epoch decay schedule remains as the BASE; the ISV + multiplier is the adaptive _curriculum modulator_ on top + +### 3.6 Reset semantics + +All 15 new slots: FoldReset (sentinel 0, Pearl A bootstraps from first +observation). Cross-fold persistence is wrong here because the +"improvement signal" should reset at each fold's cold-start. + +**Exception:** `REWARD_COMPONENT_GRAD_RATIO_BASE` slots — these accumulate +across the fold and reset to sentinel at fold boundary. The other slots +(`val_sharpe_delta`, `saboteur_engagement_rate`) need at least 2 epochs +within a fold to compute a delta; they're signal-bootstrapped so the first +epoch of each fold uses the floor (no signal yet). + +## 4. ISV slot allocation + +After SP10's last slot at 339: + +| Slot range | Purpose | Count | +|---|---|---| +| `[340..346)` | 6 component weights | 6 | +| `346` | `CURIOSITY_PRESSURE_INDEX` | 1 | +| `347` | `SABOTEUR_INTENSITY_MULT_INDEX` | 1 | +| `348` | `REWARD_WEIGHT_FLOOR_INDEX` (adaptive) | 1 | +| `349` | `CURIOSITY_BOUND_INDEX` (signal-relative) | 1 | +| `350` | `VAL_SHARPE_DELTA_EMA_INDEX` (canary) | 1 | +| `351` | `VAL_SHARPE_STD_EMA_INDEX` (canary) | 1 | +| `[352..358)` | `REWARD_COMPONENT_GRAD_RATIO_BASE` (6 canaries) | 6 | +| `358` | `SABOTEUR_ENGAGEMENT_RATE_INDEX` (canary) | 1 | +| `359` | `PNL_REWARD_MAGNITUDE_EMA_INDEX` (canary) | 1 | + +Total: **20 new slots**. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`. + +## 5. Producer kernels (4 total) + +1. `reward_subsystem_controller_kernel.cu` (the main controller, ~200 LOC) +2. `val_sharpe_delta_compute_kernel.cu` (canary — reads val sharpe history, + writes Δ + |Δ| EMAs, ~80 LOC) +3. `saboteur_engagement_compute_kernel.cu` (canary — reads per-episode Q + shift after saboteur params change, writes engagement rate, ~100 LOC) +4. `reward_component_grad_ratio_compute_kernel.cu` (canary — reads + per-component grad EMA from existing slots, normalizes to ratios, ~60 LOC) + +Plus modifications to existing kernels: +- `mse_loss_kernel.cu`, `c51_loss_kernel.cu` — read cf_weight from ISV +- Reward shaping kernels touched during audit — read all 6 component weights +- New: curiosity bonus integration in the reward path +- `gpu_experience_collector.rs` — multiply saboteur_perturbation_scale by ISV mult + +## 6. Tests + +- Unit test: synthetic z=0 → curiosity at midpoint; z>0 → curiosity drops; z<0 → curiosity max +- Unit test: weight floor ≥ WEIGHT_HARD_FLOOR always +- Unit test: saboteur_intensity ∈ [SABOTEUR_MIN, SABOTEUR_MAX] +- Contract test: 20 new slots have FoldReset entries + dispatch arms +- Layout fingerprint: ISV_TOTAL_DIM = 360 +- L40S smoke: 5-epoch run shows weights drifting from initial sentinels, + curiosity oscillating with stagnation signal, saboteur_intensity reactive + +## 7. Risks and open questions + +- **Risk 1**: Curiosity bonus could destabilize a working policy. Mitigated + by `CURIOSITY_BOUND_INDEX` = 0.3 × PnL EMA — curiosity is always at most + 30% of PnL signal magnitude. Permanent floor pattern means it never + zeros out either, so we always have some exploration. +- **Risk 2**: Per-component weight reweighting could starve a component. + Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the + least-contributing component keeps a non-trivial budget. +- **Risk 3**: Saboteur intensity rising on improvement could over-stress the + policy. Mitigated by [0.5, 2.0] hard bound and engagement-rate self- + correction (per `pearl_engagement_rate_self_correction`). +- **Risk 4**: First-fold cold-start: with only 1-2 val emits available, the + Z-score is noisy. Mitigated by Pearl A sentinel-bootstrap (first + observation replaces sentinel directly) and Pearl D Wiener-α (smooth as + data accumulates). +- **Open question**: Should the saboteur intensity also be ISV-driven on a + PER-CURRICULUM-DIMENSION basis (separate spread / fill / slippage + multipliers)? Current design uses a single scalar mult. Defer to SP11.1 + if needed after empirical results. + +## 8. Implementation phases (atomic per `feedback_no_partial_refactor`) + +Single commit: + +1. Allocate 20 new ISV slots in `sp5_isv_slots.rs` (range 340..360) +2. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`, layout fingerprint +3. Add 4 new producer kernels (controller + 3 canaries) +4. Register kernels in `build.rs` +5. Rust launchers in `gpu_dqn_trainer.rs` for all 4 new kernels +6. Wire launches into `training_loop.rs` at appropriate points +7. **Audit + migrate ALL existing reward weight constants:** + - `mse_loss_kernel.cu:318` (cf_weight = 0.3 → ISV) + - `c51_loss_kernel.cu:789` (cf_weight = 0.3 → ISV) + - Other shaping multipliers found during audit +8. Add curiosity bonus term to reward path (NEW dimension) +9. Modify `gpu_experience_collector.rs:3328` for saboteur_intensity multiplier +10. State reset registry: 20 new FoldReset entries + dispatch arms +11. Contract test passes +12. New pearl: `pearl_reward_as_controlled_subsystem.md` authored +13. MEMORY.md index entry added +14. Audit doc: Fix 39 entry with full architecture summary +15. HEALTH_DIAG line for `sp11_reward` showing all 10 outputs + improvement_z + +Total estimated: ~1300-1700 LOC across ~12 files. ~2.5-3 hours subagent work. + +## 9. Validation criteria + +After SP11 lands: + +**Smoke (5-epoch L40S):** +- HEALTH_DIAG `sp11_reward` line emits with all weights in [floor, 1.0] +- `improvement_z` evolves across epochs (not stuck at 0) +- `curiosity_pressure` oscillates with stagnation +- No regression in Fix 33-38 metrics (IQN warnings still 0, dir_entropy + still > 0.5, val_active_frac still > 0.20) + +**T10 (50-epoch full validation):** +- val sharpe trajectory shows continued IMPROVEMENT (not just stagnation + or decline) at least into ep20 +- Q-value continues to grow with val sharpe (not decoupled like ep0-14 + baseline) +- Component weight evolution visible in HEALTH_DIAG (different components + win at different training stages) + +**Specific success metric:** `val_sharpe[ep20] > val_sharpe[ep1]`. The Fix +33-38 baseline showed `ep1 > ep14` (peak at ep1, declining). SP11 +success = peak shifts to ep20+ or beyond. + +## 10. New pearl: `pearl_reward_as_controlled_subsystem` + +Authored alongside this SP. Captures the meta-principle: + +> Every degree of freedom in the reward path is an output of a unified +> controller whose inputs are improvement-rate canaries. Reward components, +> shaping multipliers, exploration bonuses, and curriculum knobs are all +> ISV-driven from a Z-score-relative improvement signal that never depends +> on hardcoded thresholds. Permanent floors prevent any component from +> being silently zeroed out; OR'd canaries prevent any single signal-failure +> from stalling the controller. This subsumes the controller-anchor pearl +> applied at the reward layer.