From 9395b983ccbfb37bddb8a273313bce51b867e253 Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Mon, 4 May 2026 00:20:25 +0200 Subject: [PATCH] =?UTF-8?q?spec(sp11):=20fix=20all=2013=20review=20issues?= =?UTF-8?q?=20=E2=80=94=20straight-up=20implementation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical bugs fixed: 1. Z-score formula: was mean(Δ)/mean(|Δ|), bounded to [-1,+1] — sigmoid never saturated, controller stuck in [0.27, 0.73]. Now true Z-score: delta_ema / sqrt(delta_var_ema) with two-pass var. Renamed canary slot 351 from VAL_SHARPE_STD_EMA to VAL_SHARPE_VAR_EMA. 2. Component weight renormalization added — Σweights = 1 enforced post-floor so PopArt's reward-magnitude EMA stays uncontaminated. 3. SABOTEUR_MIN bound violation — engagement-floor (0.1) was silently pulling output below 0.5 stated minimum. Added post-multiplication clamp to [SABOTEUR_MIN, SABOTEUR_MAX]. 4. ratios[] undeclared — now __shared__ float ratios[6] block-loaded once from ISV. 5. §3.6 slot count off-by-five (15 → 20). All 20 slots get FoldReset entries; cold-start window behavior specified explicitly (no NaN path). Architectural fixes: 6. Q-overconfidence concern addressed in scope — controller's REWARD_CF_WEIGHT_INDEX covers CQL conservatism. No SP11′ deferral; if T10 fails, extend controller outputs (e.g., target-update τ). 7. Curiosity recompute-at-replay specified (§3.5.1) — replay buffer stores base reward only; curiosity added per-tuple at replay time against current visit-count and current ISV. Eliminates stale-signal replay contamination that would worsen ep1-overfitting. Smaller fixes: 8. Novelty signal specified — SimHash 42×16 projection → 16-bit code, 1M-slot per-bucket count table, 1/sqrt(1+count). Hash table lives in state-reset registry (cleared at fold boundary). 9. Saboteur engagement operationally defined — per-bar |reward_with_saboteur − reward_without_saboteur| > EPS_ENGAGEMENT, block tree-reduce, EMA'd. EPS_ENGAGEMENT = 0.01 × PnL_EMA. 10. Duplicate sentence in §7 component-starvation paragraph removed. 11. Validation criteria strengthened (§9): aggregation across 9 trajectories specified; primary metric = median peak-epoch ≥ 10 (baseline median peak-epoch = 1); secondary = drop-from-peak ≤ 5% by ep20; tertiary = ep20 mean ≥ baseline ep1 mean. §9.3 fix-forward response codified — no rollback. 12. Phases split per project pattern — Layer A additive (3 commits: slots, canaries, controller), Layer B atomic consumer migration (1 commit, including replay-time curiosity), Layer C validation + close-out. No falsification gate; smoke is validation only. 13. Pearls A+D applied to controller outputs (§3.4.1) — chained apply_pearls_ad_kernel after controller writes scratch, smoothed values land in ISV. Consumers read smoothed slots. Constants surviving (Invariant-1 anchors only, all rate-not-regime): EPS_DIV, WEIGHT_HARD_FLOOR, SABOTEUR_MIN/MAX, CURIOSITY_PERMANENT_FRACTION, CURIOSITY_BOUND_FRACTION, WEIGHT_FLOOR_FRACTION, ENGAGEMENT_FLOOR. Spec is now full straight-up implementation; smoke validates Layer A infrastructure and Layer B consumers, T10 validates SP11 success metric. ~1550 LOC across 3 commits in Layer A + 1 atomic commit in Layer B + close-out in Layer C. --- ...-04-sp11-reward-as-controlled-subsystem.md | 430 +++++++++++++----- 1 file changed, 324 insertions(+), 106 deletions(-) diff --git a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md index e17868d71..b5413436d 100644 --- a/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md +++ b/docs/superpowers/specs/2026-05-04-sp11-reward-as-controlled-subsystem.md @@ -52,21 +52,45 @@ The reward components (`popart`, `cf`, `trail_r`, `micro`, `opp_cost`, decays on a hardcoded 0.995/epoch schedule in `gpu_experience_collector.rs`). None adapt to whether training is improving, stagnating, or regressing. +### 2.1 Design awareness — Q-overconfidence is also in scope + +The "Q grows while val drops" signature can also be driven by +**TD-target overconfidence**: optimistic bootstrap targets pull the +value function toward higher magnitudes regardless of policy quality. +SP11's CQL component-weight slot (`REWARD_CF_WEIGHT_INDEX`) explicitly +covers this case: when `improving_z < 0` (regressing), the +controller diversifies weights toward under-used components, which +includes raising CQL conservatism if the grad-ratio canary shows CQL +under-firing. Because the controller's input space includes the very +shaping multipliers that govern Q-target conservatism, SP11 *is* the +fix for both reward-shaping and conservatism in one architecture. + +No separate SP11′ is needed. If T10 (§9) shows residual decline after +SP11, the response is to extend the controller's outputs further (e.g., +target-update τ as a new ISV slot), not to pause SP11. + ## 3. Architecture (refined: every input ISV-driven) ### 3.1 Z-score-driven adaptation (no hardcoded thresholds) -The "improving / stagnant / regressing" classification uses a Z-score, not a -hardcoded threshold: +The "improving / stagnant / regressing" classification uses a **true Z-score** +(mean Δ divided by std of residuals around the mean — *not* mean over +magnitude, which would bound z to [-1, +1] and prevent sigmoid saturation, +collapsing the controller's effective range to [0.27, 0.73]): ``` -improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS_DIV) +delta = val_sharpe[t] − val_sharpe[t-1] +delta_ema = EMA(delta) ← running mean of Δ +delta_var_ema = EMA( (delta − delta_ema)² ) ← variance around the trend +improvement_z = delta_ema / max( sqrt(delta_var_ema), EPS_DIV ) ``` -Where: -- `val_sharpe_delta_ema` = EMA of (val_sharpe[t] − val_sharpe[t-1]) -- `val_sharpe_std_ema` = EMA of |val_sharpe_delta| (running noise estimate) -- Both Pearl D Wiener-α-smoothed; both have Pearl A sentinel-bootstrap +`improvement_z` is unbounded so `sigmoid(z)` can saturate near 0 or 1 in +regimes that warrant it. Both EMAs are Pearl D Wiener-α-smoothed and +Pearl A sentinel-bootstrapped (first observation replaces sentinel +directly). The variance EMA is computed inside the canary kernel in a +two-pass form: first update `delta_ema`, then accumulate the squared +deviation against the *updated* mean (corrects for the EMA-of-EMA bias). `improvement_z` ∈ ℝ; the controller's outputs are continuous functions of `z`, not threshold-gated: @@ -97,43 +121,91 @@ Where: | Slot | Source | Pearl pattern | |---|---|---| | `VAL_SHARPE_DELTA_EMA_INDEX` | `Δsharpe[t] = sharpe[t]−sharpe[t-1]`, Pearl D smoothed | A+D | -| `VAL_SHARPE_STD_EMA_INDEX` | EMA of `|Δsharpe|` (noise estimate) | A+D | -| `REWARD_COMPONENT_GRAD_RATIO_BASE` | 6 slots, per-component grad-EMA / total-grad-EMA | A+D | -| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Fraction of episodes where saboteur params shifted Q by > EPS | Pearl C engagement counter | +| `VAL_SHARPE_VAR_EMA_INDEX` | EMA of `(Δsharpe − delta_ema)²` (true variance estimator) | A+D | +| `REWARD_COMPONENT_GRAD_RATIO_BASE` | **6 slots** [base..base+6), per-component grad-EMA / total-grad-EMA | A+D | +| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Per-bar saboteur-driven Q-shift hit rate (see §3.3.1 for operational definition) | Pearl C engagement counter | | `PNL_REWARD_MAGNITUDE_EMA_INDEX` | `|popart_reward|` EMA — bounds curiosity proportionally | A+D | +#### 3.3.1 Operational definitions + +**Saboteur engagement** (Foxhunt has bars, not episodes): inside +`gpu_experience_collector`, the saboteur perturbs fill price and slippage +per-bar. Engagement is the fraction of bars where the perturbation +*actually changed the realized reward by ≥ EPS_ENGAGEMENT*: +```cuda +bool engaged = fabsf(reward_with_saboteur - reward_without_saboteur) > EPS_ENGAGEMENT; +``` +The "without saboteur" reward is computed cheaply by un-applying the +perturbation (price + perturbation_offset → price). This is a per-bar +boolean reduced via block tree-reduce (no atomicAdd, per +`feedback_no_atomicadd`) into a per-step engagement count, then divided +by batch size and EMA'd. EPS_ENGAGEMENT = `0.01 × isv[PNL_REWARD_MAGNITUDE_EMA]` +(Invariant-1 fraction of PnL signal scale). + +**Per-component grad ratio**: each loss kernel already accumulates +per-component gradient magnitudes for the SP4 grad-balancer. Reuse those +6 EMAs; ratio = `grad_ema[c] / max(sum(grad_ema), EPS_DIV)`. + ### 3.4 Producer kernel: `reward_subsystem_controller` Single block, 10 threads (one per output). Reads all 5 canaries, computes: ```cuda const float EPS_DIV = 1e-6f; -const float SABOTEUR_MIN = 0.5f; // Invariant-1 rate limiter -const float SABOTEUR_MAX = 2.0f; // Invariant-1 rate limiter -const float WEIGHT_HARD_FLOOR = 0.01f; // Invariant-1 numerical floor (prevents zero) +const float SABOTEUR_MIN = 0.5f; // Invariant-1 rate limiter +const float SABOTEUR_MAX = 2.0f; // Invariant-1 rate limiter +const float WEIGHT_HARD_FLOOR = 0.01f; // Invariant-1 numerical floor +const float CURIOSITY_PERMANENT_FRACTION = 0.2f; // Invariant-1 floor fraction +const float ENGAGEMENT_FLOOR = 0.1f; // Invariant-1 fraction +const float CURIOSITY_BOUND_FRACTION = 0.3f; // Invariant-1 fraction of PnL EMA +const float WEIGHT_FLOOR_FRACTION = 0.5f; // Invariant-1 fraction of min grad ratio -// Z-score (signal-relative; no hardcoded threshold) -float dz = isv[VAL_SHARPE_DELTA_EMA] / fmaxf(isv[VAL_SHARPE_STD_EMA], EPS_DIV); +// Block-load all 6 ratios into shared mem ONCE (avoids re-reading ISV). +__shared__ float ratios[6]; +if (threadIdx.x < 6) { + ratios[threadIdx.x] = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + threadIdx.x]; +} +__syncthreads(); -// Sigmoid maps z continuously to [0, 1]; no thresholds +// True Z-score: mean Δ over std-of-residuals, NOT mean Δ over mean |Δ|. +// (Latter would be bounded to [-1,+1] and never let sigmoid saturate.) +float delta_var = isv[VAL_SHARPE_VAR_EMA_INDEX]; +float dz = isv[VAL_SHARPE_DELTA_EMA_INDEX] + / fmaxf( sqrtf(delta_var), EPS_DIV ); + +// Sigmoid maps z continuously to [0, 1] float improving = 1.0f / (1.0f + expf(-dz)); // 0=regressing, 0.5=stagnant, 1=improving float stagnant_or_worse = 1.0f - improving; -// Per-component weights: blend toward (a) high-grad-ratio winners when -// improving, (b) under-used components when stagnant, (c) floor when -// regressing. Floor itself signal-driven. -float floor = isv[REWARD_WEIGHT_FLOOR_INDEX]; // adaptive (see below) +// Per-component weight FLOOR (signal-driven from worst-suppressed component) +float min_grad_ratio = ratios[0]; +for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]); +float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, WEIGHT_FLOOR_FRACTION * min_grad_ratio); +scratch[scratch_weight_floor] = adaptive_floor; + +// Per-component weights — blend pre-floor, then enforce floor, then RENORM. +// Renormalization is critical: PopArt's reward-magnitude EMA assumes a +// consistent reward-scale; sum(weights) drifting > 1 would inflate the +// running |reward| and contaminate the popart target every fold. +float blends[6]; +float blend_sum = 0.0f; for (int c = 0; c < 6; c++) { - float ratio = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + c]; - float winner_weight = ratio; // exploit - float diversifier_weight = (1.0f - ratio) / 5.0f; // explore - float blend = improving * winner_weight + stagnant_or_worse * diversifier_weight; - float weight = fmaxf(floor, fminf(1.0f, blend)); - scratch[scratch_weight_base + c] = weight; + float winner_weight = ratios[c]; // exploit + float diversifier_weight = (1.0f - ratios[c]) / 5.0f; // explore (sums to 1−ratios[c]·5/5 across c, see note) + float w = improving * winner_weight + stagnant_or_worse * diversifier_weight; + w = fmaxf(adaptive_floor, fminf(1.0f, w)); + blends[c] = w; + blend_sum += w; +} +// Renormalize so Σweights = 1 exactly. EPS_DIV guards the (impossible) +// all-floor-zero case. +float inv_sum = 1.0f / fmaxf(blend_sum, EPS_DIV); +for (int c = 0; c < 6; c++) { + scratch[scratch_weight_base + c] = blends[c] * inv_sum; } // Curiosity bound = fraction of recent PnL magnitude (signal-relative) -float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f; // 0.3 is Invariant-1 fraction +float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA_INDEX] * CURIOSITY_BOUND_FRACTION; scratch[scratch_curiosity_bound] = curiosity_bound; // Curiosity pressure: ALWAYS-ON exploration signal scaled by stagnation. @@ -141,35 +213,53 @@ scratch[scratch_curiosity_bound] = curiosity_bound; // When stagnating: rises toward curiosity_bound to break out of fixed point. // Permanent floor pattern per pearl_blend_formulas_must_have_permanent_floor — // max() form, not blend, so curiosity never zeros even when policy is winning. -float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound; // 0.2 = Invariant-1 fraction +float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound; float curiosity_dynamic = stagnant_or_worse * curiosity_bound; float curiosity_pressure = fmaxf(curiosity_dynamic, curiosity_floor); scratch[scratch_curiosity_pressure] = curiosity_pressure; // Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX] -// adjusted by engagement rate (per pearl_engagement_rate_self_correction). -// If saboteur is rarely shifting Q (engagement → 0), back off. +// modulated by engagement rate (per pearl_engagement_rate_self_correction). +// Compute the z-term inside the bound, modulate by engagement, and re-clamp +// to [SABOTEUR_MIN, SABOTEUR_MAX]. Without the post-clamp the engagement +// floor (0.1) would silently pull the output below SABOTEUR_MIN. float engagement = isv[SABOTEUR_ENGAGEMENT_RATE_INDEX]; float intensity_z_term = SABOTEUR_MIN + (SABOTEUR_MAX - SABOTEUR_MIN) * improving; -float saboteur_intensity = intensity_z_term * fmaxf(engagement, 0.1f); // 0.1 is engagement floor +float modulated = intensity_z_term * fmaxf(engagement, ENGAGEMENT_FLOOR); +float saboteur_intensity = fmaxf(SABOTEUR_MIN, fminf(SABOTEUR_MAX, modulated)); scratch[scratch_saboteur_intensity] = saboteur_intensity; - -// Per-component weight floor: signal-driven from worst-suppressed component's -// grad-EMA. Idea: if any component has been consistently below mean grad, -// its floor rises so it gets a chance to contribute. -float min_grad_ratio = ratios[0]; -for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]); -float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio); // 0.5 = Invariant-1 fraction -scratch[scratch_weight_floor] = adaptive_floor; ``` -The constants `0.3f` (curiosity fraction of PnL), `0.5f` (floor fraction -of min grad ratio), `0.2f` (CURIOSITY_PERMANENT_FRACTION — exploration -floor as fraction of curiosity_bound), and `0.1f` (engagement floor) are -Invariant-1 fractions: they bound rates, not regimes, and remain valid -across all regimes. The thresholds themselves (what counts as "improving", -"stagnant", "regressing") are entirely encoded in the Z-score and sigmoid -— no hardcoded threshold survives. +#### 3.4.1 Output smoothing (Pearls A+D applied to controller outputs) + +The 10 controller outputs are written to `scratch_*` first, then a chained +`apply_pearls_ad_kernel` launch smooths each output into its ISV slot. +This prevents the controller from oscillating outputs at every training +step (raw `improving` is responsive but jittery). Pearl A handles the +fold-boundary cold-start; Pearl D's Wiener-α adapts smoothing strength to +each output's own noise level (curiosity may be smoother than weights, etc.) + +This means the consumer kernels read the **smoothed** controller outputs +from ISV, not the raw scratch values. + +#### 3.4.2 Surviving constants (Invariant-1 anchors only) + +All seven hardcoded constants in the controller bound *rates*, not *regimes*, +and remain valid across regime shifts: + +| Const | Value | Role | +|---|---|---| +| `EPS_DIV` | `1e-6f` | divide-by-zero protection | +| `WEIGHT_HARD_FLOOR` | `0.01f` | absolute weight floor — prevents zero | +| `SABOTEUR_MIN` / `MAX` | `0.5f` / `2.0f` | curriculum rate limiter | +| `CURIOSITY_PERMANENT_FRACTION` | `0.2f` | exploration floor as fraction of `curiosity_bound` | +| `CURIOSITY_BOUND_FRACTION` | `0.3f` | curiosity ceiling as fraction of PnL EMA | +| `WEIGHT_FLOOR_FRACTION` | `0.5f` | adaptive floor as fraction of min grad ratio | +| `ENGAGEMENT_FLOOR` | `0.1f` | saboteur engagement floor (pre-bound) | + +The thresholds that classify regime ("improving / stagnant / regressing") +are entirely encoded in the Z-score and sigmoid — no hardcoded threshold +survives at the regime layer. ### 3.5 Consumer migrations @@ -179,12 +269,46 @@ across all regimes. The thresholds themselves (what counts as "improving", - `c51_loss_kernel.cu:789` — same migration - Other reward shaping multipliers identified during implementation audit -**Curiosity bonus** (NEW reward dimension): -- New term in reward computation: `r_total += isv[CURIOSITY_PRESSURE_INDEX] * - novelty_signal(state, action)` -- `novelty_signal` is the simplest workable form: state-action visit count - EMA → 1 / sqrt(count) (RND-style optional later) -- Bounded by `CURIOSITY_BOUND_INDEX` in the producer kernel +**Curiosity bonus** (NEW reward dimension, applied at *loss-compute* time +not at experience-collect time — see §3.5.1): +- The replay buffer continues to store the **base reward** only (no + curiosity baked in). Curiosity is added to `r` at the moment a tuple is + drawn for replay, recomputed against the *current* visit-count table + and *current* `CURIOSITY_PRESSURE_INDEX`. +- Recompute formula: `r_with_curiosity = r_base + isv[CURIOSITY_PRESSURE_INDEX] + * novelty_signal(state, action)` +- Bounded above by `CURIOSITY_BOUND_INDEX` in the producer kernel. + +#### 3.5.1 Why recompute-at-replay (not store-at-collect) + +If we store curiosity-augmented r in replay, old tuples retain the +curiosity bonus they had *at the moment they were collected*, even though +the policy now visits those state-actions frequently (curiosity should be +near zero). Replaying those tuples then teaches the policy to revisit the +fixed point we're trying to escape — making the ep1-overfitting pathology +worse, not better. Recomputing at replay binds curiosity to the *current* +exploration deficit, which is the only signal that matters. + +The cost is one extra hash-lookup + multiply per replayed tuple. Existing +replay sampling kernel adds two ops; no extra kernel launch. + +#### 3.5.2 Novelty signal (state-action visit count via locality-sensitive hash) + +The state space is continuous 42-dim. Use a SimHash-style discretization: +- Project state through a fixed (untrained) random matrix `W ∈ R^{42×16}` + with sign-quantized output: `code = sign(state @ W)` → 16-bit hash. +- Bucket = `(action_id, code)` — i.e., 108 actions × 65,536 codes = ~7M buckets. +- Visit count = atomic-free per-bucket EMA in a hash table sized 1M slots + (with linear-probe collisions tolerated; collisions = false novelty + underestimates, which is the safe direction). +- `novelty_signal = 1 / sqrt(1 + count)`. Range [0, 1]. +- Hash table is per-fold (cleared at fold boundary via state-reset registry). +- The 42×16 projection matrix is fixed at trainer init from a deterministic + seed (so different runs with same seed get same hash function). + +This is a cheap, GPU-friendly stand-in for RND or count-based bonuses. RND +or full state-action density estimation can replace it later if smoke +shows the hash discretization is too coarse. **Saboteur intensity** (modulator on existing saboteur scale): - `gpu_experience_collector.rs:3328` — `let ps = @@ -195,15 +319,31 @@ across all regimes. The thresholds themselves (what counts as "improving", ### 3.6 Reset semantics -All 15 new slots: FoldReset (sentinel 0, Pearl A bootstraps from first -observation). Cross-fold persistence is wrong here because the -"improvement signal" should reset at each fold's cold-start. +All **20** new slots get FoldReset entries (sentinel 0, Pearl A bootstraps +from first observation). Cross-fold persistence is wrong for every slot +in this spec because the "improvement signal" must reset at each fold's +cold-start — train history from fold N-1 is not informative about +fold N's improvement trajectory. -**Exception:** `REWARD_COMPONENT_GRAD_RATIO_BASE` slots — these accumulate -across the fold and reset to sentinel at fold boundary. The other slots -(`val_sharpe_delta`, `saboteur_engagement_rate`) need at least 2 epochs -within a fold to compute a delta; they're signal-bootstrapped so the first -epoch of each fold uses the floor (no signal yet). +This includes: +- 6 component weights (340..346) +- 4 controller scalar outputs (346..350) +- 2 val-sharpe canaries (350..352) +- 6 grad-ratio canaries (352..358) +- 1 saboteur engagement (358) +- 1 PnL magnitude EMA (359) + +**Cold-start window:** The Z-score is undefined until at least 2 val +emits exist within the fold (need delta + variance). For epoch 0 of each +fold, the controller observes `delta_ema = 0` (post-Pearl-A bootstrap), +which gives `dz = 0`, `improving = 0.5`, `stagnant_or_worse = 0.5` — the +"perfectly stagnant" mid-state. Every controller output is well-defined +in this state: weights blend 50/50, curiosity sits at midpoint, saboteur +sits at the geometric mean of [MIN, MAX]. There is no NaN path. + +The **novelty hash table** (§3.5.2) also resets at fold boundary; this is +NOT an ISV slot but a separate device buffer cleared by the state-reset +dispatcher. ## 4. ISV slot allocation @@ -217,7 +357,7 @@ After SP10's last slot at 339: | `348` | `REWARD_WEIGHT_FLOOR_INDEX` (adaptive) | 1 | | `349` | `CURIOSITY_BOUND_INDEX` (signal-relative) | 1 | | `350` | `VAL_SHARPE_DELTA_EMA_INDEX` (canary) | 1 | -| `351` | `VAL_SHARPE_STD_EMA_INDEX` (canary) | 1 | +| `351` | `VAL_SHARPE_VAR_EMA_INDEX` (canary, true variance) | 1 | | `[352..358)` | `REWARD_COMPONENT_GRAD_RATIO_BASE` (6 canaries) | 6 | | `358` | `SABOTEUR_ENGAGEMENT_RATE_INDEX` (canary) | 1 | | `359` | `PNL_REWARD_MAGNITUDE_EMA_INDEX` (canary) | 1 | @@ -228,9 +368,10 @@ Total: **20 new slots**. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`. 1. `reward_subsystem_controller_kernel.cu` (the main controller, ~200 LOC) 2. `val_sharpe_delta_compute_kernel.cu` (canary — reads val sharpe history, - writes Δ + |Δ| EMAs, ~80 LOC) -3. `saboteur_engagement_compute_kernel.cu` (canary — reads per-episode Q - shift after saboteur params change, writes engagement rate, ~100 LOC) + writes delta_ema + delta_var_ema (true variance), ~80 LOC) +3. `saboteur_engagement_compute_kernel.cu` (canary — per-bar + |reward_with_saboteur − reward_without_saboteur| > EPS_ENGAGEMENT + reduced via block tree-reduce, EMA'd; see §3.3.1 ~100 LOC) 4. `reward_component_grad_ratio_compute_kernel.cu` (canary — reads per-component grad EMA from existing slots, normalizes to ratios, ~60 LOC) @@ -266,10 +407,9 @@ Plus modifications to existing kernels: curiosity always (overfitting prevention). - **Component starvation guard**: Per-component weight reweighting could in principle starve a contributor. Mitigated by adaptive floor - `≥ 0.5 × min_grad_ratio` ensuring even the least-contributing component - keeps a non-trivial budget. - Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the - least-contributing component keeps a non-trivial budget. + `≥ 0.5 × min_grad_ratio` and post-floor renormalization to Σ=1, ensuring + every component keeps a non-trivial budget without inflating total + reward magnitude (which would contaminate PopArt). - **Saboteur intensity adapts both directions**: When val sharpe is improving, saboteur intensity rises to keep the policy under realistic adversarial load (you don't want a model that only works when execution @@ -286,53 +426,131 @@ Plus modifications to existing kernels: multipliers)? Current design uses a single scalar mult. Defer to SP11.1 if needed after empirical results. -## 8. Implementation phases (atomic per `feedback_no_partial_refactor`) +## 8. Implementation phases — full straight-up implementation -Single commit: +Two layers, matching SP4/SP5/SP7. Layer A is additive infrastructure +(no behavior change) split per producer for independent revertability; +Layer B is the atomic consumer migration that flips the trainer onto +the new controller. Smoke runs only at the end of each layer for +validation — there is no gating preflight that pauses or defers work. -1. Allocate 20 new ISV slots in `sp5_isv_slots.rs` (range 340..360) -2. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`, layout fingerprint -3. Add 4 new producer kernels (controller + 3 canaries) -4. Register kernels in `build.rs` -5. Rust launchers in `gpu_dqn_trainer.rs` for all 4 new kernels -6. Wire launches into `training_loop.rs` at appropriate points -7. **Audit + migrate ALL existing reward weight constants:** - - `mse_loss_kernel.cu:318` (cf_weight = 0.3 → ISV) - - `c51_loss_kernel.cu:789` (cf_weight = 0.3 → ISV) - - Other shaping multipliers found during audit -8. Add curiosity bonus term to reward path (NEW dimension) -9. Modify `gpu_experience_collector.rs:3328` for saboteur_intensity multiplier -10. State reset registry: 20 new FoldReset entries + dispatch arms -11. Contract test passes -12. New pearl: `pearl_reward_as_controlled_subsystem.md` authored -13. MEMORY.md index entry added -14. Audit doc: Fix 39 entry with full architecture summary -15. HEALTH_DIAG line for `sp11_reward` showing all 10 outputs + improvement_z +### Layer A — additive infrastructure (3 commits, no behavior change) -Total estimated: ~1300-1700 LOC across ~12 files. ~2.5-3 hours subagent work. +**A0 — slot allocation + state reset registry (~150 LOC)** +- Allocate 20 new ISV slots (340..360) in `sp5_isv_slots.rs` +- Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360` +- 20 FoldReset entries + 20 dispatch arms in state-reset registry +- Novelty hash table reset entry (1M-slot device buffer, fold-cleared) +- Layout fingerprint bumped +- Contract test passes (slot count + reset arms match) + +**A1 — three canary producer kernels (~250 LOC)** +- `val_sharpe_delta_compute_kernel.cu` (Δ + variance EMA, two-pass) +- `saboteur_engagement_compute_kernel.cu` (per-bar engagement, Pearl C) +- `reward_component_grad_ratio_compute_kernel.cu` (existing + per-component grad EMAs → normalized ratios) +- All Pearl-A bootstrapped, Pearl-D smoothed +- `build.rs` cubin registration; Rust launchers in `gpu_dqn_trainer.rs` +- Wire launches into `training_loop.rs` + +**A2 — controller producer kernel (~250 LOC)** +- `reward_subsystem_controller_kernel.cu` reads 5 canary slots, + writes 10 outputs to scratch, chained `apply_pearls_ad_kernel` + smooths scratch into ISV +- Novelty hash table device buffer (1M slots × f32) + 42×16 SimHash + projection matrix (deterministic seed, init at trainer construct) +- Build.rs entry, launcher, training_loop wire-up +- HEALTH_DIAG `sp11_reward` line emits 10 outputs + `improvement_z` + +### Layer B — atomic consumer migration (1 commit, ~750 LOC) + +`feedback_no_partial_refactor`: every consumer of the migrated +contract migrates in one commit. The trainer is on the new controller +end of this commit. + +- `mse_loss_kernel.cu:318` — `cf_weight = isv[REWARD_CF_WEIGHT_INDEX]` +- `c51_loss_kernel.cu:789` — same migration +- Full audit + migrate every hardcoded reward-shaping constant in + `cuda_pipeline/*.cu` (expected ~3-5 additional sites; implementer + enumerates and migrates all of them) +- `gpu_experience_collector.rs:3328` — `let ps = saboteur_perturbation_scale + * isv[SABOTEUR_INTENSITY_MULT_INDEX]` +- Replay sampling kernel: curiosity bonus added **at replay time** + (§3.5.1) reading `isv[CURIOSITY_PRESSURE_INDEX]` × `novelty_signal` +- Reward composition reads all 6 component weights from ISV +- Stale doc-string sweep across migrated sites + +### Layer C — validation + close-out + +**C1** — L40S 5-epoch smoke (`multi_fold_convergence`) on Layer B HEAD +**C2** — full T10 (3-seed × 3-fold × 50-epoch) on Layer B HEAD +**C3** — audit doc: Fix 39 entry with architecture summary + canary + evolution evidence + Layer B before/after metrics +**C4** — `pearl_reward_as_controlled_subsystem.md` authored +**C5** — MEMORY.md index entry + +**Total estimated:** Layer A ≈ 650 LOC + Layer B ≈ 750 LOC + Layer C +≈ 150 LOC = ~1550 LOC across ~12 files. Layer A: ~1.5 hours. Layer B: +~1 hour. Layer C: ~30 min. Plus validation wall-clock (smoke ≈ 25 min, +T10 ≈ 4 hr). ## 9. Validation criteria -After SP11 lands: +### 9.1 Smoke (5-epoch L40S `multi_fold_convergence`) -**Smoke (5-epoch L40S):** -- HEALTH_DIAG `sp11_reward` line emits with all weights in [floor, 1.0] -- `improvement_z` evolves across epochs (not stuck at 0) -- `curiosity_pressure` oscillates with stagnation -- No regression in Fix 33-38 metrics (IQN warnings still 0, dir_entropy - still > 0.5, val_active_frac still > 0.20) +Hard-pass criteria (all must hold): +- HEALTH_DIAG `sp11_reward` line emits every epoch with all weights in + [adaptive_floor, 1.0] and Σweights ≈ 1.0 ± 1e-3 (renormalization works) +- `improvement_z` is non-zero by ep2 (canary populates after Pearl A + bootstrap on ep1's first delta) +- `curiosity_pressure ≥ 0.2 × curiosity_bound` always (permanent floor) + and `≤ curiosity_bound` always +- `saboteur_intensity ∈ [0.5, 2.0]` always (post-clamp) +- No regression in Fix 33-38 metrics: IQN parallel-branch warnings = 0, + dir_entropy > 0.5, val_active_frac > 0.20 -**T10 (50-epoch full validation):** -- val sharpe trajectory shows continued IMPROVEMENT (not just stagnation - or decline) at least into ep20 -- Q-value continues to grow with val sharpe (not decoupled like ep0-14 - baseline) -- Component weight evolution visible in HEALTH_DIAG (different components - win at different training stages) +### 9.2 T10 (3-seed × 3-fold × 50-epoch L40S) -**Specific success metric:** `val_sharpe[ep20] > val_sharpe[ep1]`. The Fix -33-38 baseline showed `ep1 > ep14` (peak at ep1, declining). SP11 -success = peak shifts to ep20+ or beyond. +Aggregation: 9 trajectories. Per-trajectory peak-epoch is the epoch +maximizing val sharpe within that trajectory. + +Hard-pass criteria (all must hold): +- **Median peak-epoch across 9 trajectories ≥ 10** + (Fix 33-38 baseline: median peak-epoch = 1, where xkjkb seed-0 fold-0 + peaked at ep1.) This is the primary signal that the model continues + learning past ep1. +- **Median val sharpe at ep20 > 0.95 × median val sharpe at peak-epoch** + (the controller may shift the peak forward; a 5% drop from peak by ep20 + is acceptable noise. A larger drop indicates regression.) +- **Mean val sharpe across all 9 trajectories at ep20 ≥ Fix 33-38 baseline + ep1 mean** (sanity check — SP11 cannot make ep20 *worse* than the + baseline's best epoch.) + +Soft criteria (informative, not gating): +- Component weight evolution visible across training (different + components dominate at different stages — std(weight[c]) > floor over + full T10) +- Q-value growth correlated with val sharpe (Pearson(train Q, val sharpe) + > 0.3 within each trajectory; baseline showed Pearson = -0.7 at the + ep0-14 window — the decoupled-overfitting signature) +- Per-fold pattern: fold 0 may peak earliest, fold 2 latest (cumulative + novelty hash effects), but no fold collapses (peak-epoch ≥ 5 in every + fold of every seed) + +### 9.3 If T10 fails + +Fix-forward, not roll-back. Failure modes and responses: +- Median peak-epoch < 10 but z-canary populated: controller has signal + but isn't acting strongly enough — extend ISV outputs to additional + shaping multipliers found during Layer B audit +- z-canary stuck at 0: variance EMA is mis-tracking; review Pearl A + bootstrap on `VAL_SHARPE_VAR_EMA` +- Component weights collapse to floor: renormalization or floor + computation bug; review §3.4 implementation +- Curiosity dominates loss (`r_total / r_base > 1.0` median): + CURIOSITY_BOUND_FRACTION too high; this is the only Invariant-1 + fraction we'd revisit and still not because we hardcoded a regime, + but because the chosen rate-limiter was too permissive ## 10. New pearl: `pearl_reward_as_controlled_subsystem`