spec(sp11): fix all 13 review issues — straight-up implementation
Critical bugs fixed:
1. Z-score formula: was mean(Δ)/mean(|Δ|), bounded to [-1,+1] — sigmoid
never saturated, controller stuck in [0.27, 0.73]. Now true Z-score:
delta_ema / sqrt(delta_var_ema) with two-pass var. Renamed canary
slot 351 from VAL_SHARPE_STD_EMA to VAL_SHARPE_VAR_EMA.
2. Component weight renormalization added — Σweights = 1 enforced
post-floor so PopArt's reward-magnitude EMA stays uncontaminated.
3. SABOTEUR_MIN bound violation — engagement-floor (0.1) was silently
pulling output below 0.5 stated minimum. Added post-multiplication
clamp to [SABOTEUR_MIN, SABOTEUR_MAX].
4. ratios[] undeclared — now __shared__ float ratios[6] block-loaded
once from ISV.
5. §3.6 slot count off-by-five (15 → 20). All 20 slots get FoldReset
entries; cold-start window behavior specified explicitly (no NaN path).
Architectural fixes:
6. Q-overconfidence concern addressed in scope — controller's
REWARD_CF_WEIGHT_INDEX covers CQL conservatism. No SP11′ deferral;
if T10 fails, extend controller outputs (e.g., target-update τ).
7. Curiosity recompute-at-replay specified (§3.5.1) — replay buffer
stores base reward only; curiosity added per-tuple at replay time
against current visit-count and current ISV. Eliminates stale-signal
replay contamination that would worsen ep1-overfitting.
Smaller fixes:
8. Novelty signal specified — SimHash 42×16 projection → 16-bit code,
1M-slot per-bucket count table, 1/sqrt(1+count). Hash table lives
in state-reset registry (cleared at fold boundary).
9. Saboteur engagement operationally defined — per-bar
|reward_with_saboteur − reward_without_saboteur| > EPS_ENGAGEMENT,
block tree-reduce, EMA'd. EPS_ENGAGEMENT = 0.01 × PnL_EMA.
10. Duplicate sentence in §7 component-starvation paragraph removed.
11. Validation criteria strengthened (§9): aggregation across 9
trajectories specified; primary metric = median peak-epoch ≥ 10
(baseline median peak-epoch = 1); secondary = drop-from-peak ≤ 5%
by ep20; tertiary = ep20 mean ≥ baseline ep1 mean. §9.3 fix-forward
response codified — no rollback.
12. Phases split per project pattern — Layer A additive (3 commits:
slots, canaries, controller), Layer B atomic consumer migration
(1 commit, including replay-time curiosity), Layer C validation +
close-out. No falsification gate; smoke is validation only.
13. Pearls A+D applied to controller outputs (§3.4.1) — chained
apply_pearls_ad_kernel after controller writes scratch, smoothed
values land in ISV. Consumers read smoothed slots.
Constants surviving (Invariant-1 anchors only, all rate-not-regime):
EPS_DIV, WEIGHT_HARD_FLOOR, SABOTEUR_MIN/MAX, CURIOSITY_PERMANENT_FRACTION,
CURIOSITY_BOUND_FRACTION, WEIGHT_FLOOR_FRACTION, ENGAGEMENT_FLOOR.
Spec is now full straight-up implementation; smoke validates Layer A
infrastructure and Layer B consumers, T10 validates SP11 success metric.
~1550 LOC across 3 commits in Layer A + 1 atomic commit in Layer B +
close-out in Layer C.
This commit is contained in:
jgrusewski
@@ -52,21 +52,45 @@ The reward components (`popart`, `cf`, `trail_r`, `micro`, `opp_cost`,
|
||||
decays on a hardcoded 0.995/epoch schedule in `gpu_experience_collector.rs`).
|
||||
None adapt to whether training is improving, stagnating, or regressing.
|
||||
|
||||
### 2.1 Design awareness — Q-overconfidence is also in scope
|
||||
|
||||
The "Q grows while val drops" signature can also be driven by
|
||||
**TD-target overconfidence**: optimistic bootstrap targets pull the
|
||||
value function toward higher magnitudes regardless of policy quality.
|
||||
SP11's CQL component-weight slot (`REWARD_CF_WEIGHT_INDEX`) explicitly
|
||||
covers this case: when `improving_z < 0` (regressing), the
|
||||
controller diversifies weights toward under-used components, which
|
||||
includes raising CQL conservatism if the grad-ratio canary shows CQL
|
||||
under-firing. Because the controller's input space includes the very
|
||||
shaping multipliers that govern Q-target conservatism, SP11 *is* the
|
||||
fix for both reward-shaping and conservatism in one architecture.
|
||||
|
||||
No separate SP11′ is needed. If T10 (§9) shows residual decline after
|
||||
SP11, the response is to extend the controller's outputs further (e.g.,
|
||||
target-update τ as a new ISV slot), not to pause SP11.
|
||||
|
||||
## 3. Architecture (refined: every input ISV-driven)
|
||||
|
||||
### 3.1 Z-score-driven adaptation (no hardcoded thresholds)
|
||||
|
||||
The "improving / stagnant / regressing" classification uses a Z-score, not a
|
||||
hardcoded threshold:
|
||||
The "improving / stagnant / regressing" classification uses a **true Z-score**
|
||||
(mean Δ divided by std of residuals around the mean — *not* mean over
|
||||
magnitude, which would bound z to [-1, +1] and prevent sigmoid saturation,
|
||||
collapsing the controller's effective range to [0.27, 0.73]):
|
||||
|
||||
```
|
||||
improvement_z = val_sharpe_delta_ema / max(val_sharpe_std_ema, EPS_DIV)
|
||||
delta = val_sharpe[t] − val_sharpe[t-1]
|
||||
delta_ema = EMA(delta) ← running mean of Δ
|
||||
delta_var_ema = EMA( (delta − delta_ema)² ) ← variance around the trend
|
||||
improvement_z = delta_ema / max( sqrt(delta_var_ema), EPS_DIV )
|
||||
```
|
||||
|
||||
Where:
|
||||
- `val_sharpe_delta_ema` = EMA of (val_sharpe[t] − val_sharpe[t-1])
|
||||
- `val_sharpe_std_ema` = EMA of |val_sharpe_delta| (running noise estimate)
|
||||
- Both Pearl D Wiener-α-smoothed; both have Pearl A sentinel-bootstrap
|
||||
`improvement_z` is unbounded so `sigmoid(z)` can saturate near 0 or 1 in
|
||||
regimes that warrant it. Both EMAs are Pearl D Wiener-α-smoothed and
|
||||
Pearl A sentinel-bootstrapped (first observation replaces sentinel
|
||||
directly). The variance EMA is computed inside the canary kernel in a
|
||||
two-pass form: first update `delta_ema`, then accumulate the squared
|
||||
deviation against the *updated* mean (corrects for the EMA-of-EMA bias).
|
||||
|
||||
`improvement_z` ∈ ℝ; the controller's outputs are continuous functions of
|
||||
`z`, not threshold-gated:
|
||||
@@ -97,43 +121,91 @@ Where:
|
||||
| Slot | Source | Pearl pattern |
|
||||
|---|---|---|
|
||||
| `VAL_SHARPE_DELTA_EMA_INDEX` | `Δsharpe[t] = sharpe[t]−sharpe[t-1]`, Pearl D smoothed | A+D |
|
||||
| `VAL_SHARPE_STD_EMA_INDEX` | EMA of `|Δsharpe|` (noise estimate) | A+D |
|
||||
| `REWARD_COMPONENT_GRAD_RATIO_BASE` | 6 slots, per-component grad-EMA / total-grad-EMA | A+D |
|
||||
| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Fraction of episodes where saboteur params shifted Q by > EPS | Pearl C engagement counter |
|
||||
| `VAL_SHARPE_VAR_EMA_INDEX` | EMA of `(Δsharpe − delta_ema)²` (true variance estimator) | A+D |
|
||||
| `REWARD_COMPONENT_GRAD_RATIO_BASE` | **6 slots** [base..base+6), per-component grad-EMA / total-grad-EMA | A+D |
|
||||
| `SABOTEUR_ENGAGEMENT_RATE_INDEX` | Per-bar saboteur-driven Q-shift hit rate (see §3.3.1 for operational definition) | Pearl C engagement counter |
|
||||
| `PNL_REWARD_MAGNITUDE_EMA_INDEX` | `|popart_reward|` EMA — bounds curiosity proportionally | A+D |
|
||||
|
||||
#### 3.3.1 Operational definitions
|
||||
|
||||
**Saboteur engagement** (Foxhunt has bars, not episodes): inside
|
||||
`gpu_experience_collector`, the saboteur perturbs fill price and slippage
|
||||
per-bar. Engagement is the fraction of bars where the perturbation
|
||||
*actually changed the realized reward by ≥ EPS_ENGAGEMENT*:
|
||||
```cuda
|
||||
bool engaged = fabsf(reward_with_saboteur - reward_without_saboteur) > EPS_ENGAGEMENT;
|
||||
```
|
||||
The "without saboteur" reward is computed cheaply by un-applying the
|
||||
perturbation (price + perturbation_offset → price). This is a per-bar
|
||||
boolean reduced via block tree-reduce (no atomicAdd, per
|
||||
`feedback_no_atomicadd`) into a per-step engagement count, then divided
|
||||
by batch size and EMA'd. EPS_ENGAGEMENT = `0.01 × isv[PNL_REWARD_MAGNITUDE_EMA]`
|
||||
(Invariant-1 fraction of PnL signal scale).
|
||||
|
||||
**Per-component grad ratio**: each loss kernel already accumulates
|
||||
per-component gradient magnitudes for the SP4 grad-balancer. Reuse those
|
||||
6 EMAs; ratio = `grad_ema[c] / max(sum(grad_ema), EPS_DIV)`.
|
||||
|
||||
### 3.4 Producer kernel: `reward_subsystem_controller`
|
||||
|
||||
Single block, 10 threads (one per output). Reads all 5 canaries, computes:
|
||||
|
||||
```cuda
|
||||
const float EPS_DIV = 1e-6f;
|
||||
const float SABOTEUR_MIN = 0.5f; // Invariant-1 rate limiter
|
||||
const float SABOTEUR_MAX = 2.0f; // Invariant-1 rate limiter
|
||||
const float WEIGHT_HARD_FLOOR = 0.01f; // Invariant-1 numerical floor (prevents zero)
|
||||
const float SABOTEUR_MIN = 0.5f; // Invariant-1 rate limiter
|
||||
const float SABOTEUR_MAX = 2.0f; // Invariant-1 rate limiter
|
||||
const float WEIGHT_HARD_FLOOR = 0.01f; // Invariant-1 numerical floor
|
||||
const float CURIOSITY_PERMANENT_FRACTION = 0.2f; // Invariant-1 floor fraction
|
||||
const float ENGAGEMENT_FLOOR = 0.1f; // Invariant-1 fraction
|
||||
const float CURIOSITY_BOUND_FRACTION = 0.3f; // Invariant-1 fraction of PnL EMA
|
||||
const float WEIGHT_FLOOR_FRACTION = 0.5f; // Invariant-1 fraction of min grad ratio
|
||||
|
||||
// Z-score (signal-relative; no hardcoded threshold)
|
||||
float dz = isv[VAL_SHARPE_DELTA_EMA] / fmaxf(isv[VAL_SHARPE_STD_EMA], EPS_DIV);
|
||||
// Block-load all 6 ratios into shared mem ONCE (avoids re-reading ISV).
|
||||
__shared__ float ratios[6];
|
||||
if (threadIdx.x < 6) {
|
||||
ratios[threadIdx.x] = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + threadIdx.x];
|
||||
}
|
||||
__syncthreads();
|
||||
|
||||
// Sigmoid maps z continuously to [0, 1]; no thresholds
|
||||
// True Z-score: mean Δ over std-of-residuals, NOT mean Δ over mean |Δ|.
|
||||
// (Latter would be bounded to [-1,+1] and never let sigmoid saturate.)
|
||||
float delta_var = isv[VAL_SHARPE_VAR_EMA_INDEX];
|
||||
float dz = isv[VAL_SHARPE_DELTA_EMA_INDEX]
|
||||
/ fmaxf( sqrtf(delta_var), EPS_DIV );
|
||||
|
||||
// Sigmoid maps z continuously to [0, 1]
|
||||
float improving = 1.0f / (1.0f + expf(-dz)); // 0=regressing, 0.5=stagnant, 1=improving
|
||||
float stagnant_or_worse = 1.0f - improving;
|
||||
|
||||
// Per-component weights: blend toward (a) high-grad-ratio winners when
|
||||
// improving, (b) under-used components when stagnant, (c) floor when
|
||||
// regressing. Floor itself signal-driven.
|
||||
float floor = isv[REWARD_WEIGHT_FLOOR_INDEX]; // adaptive (see below)
|
||||
// Per-component weight FLOOR (signal-driven from worst-suppressed component)
|
||||
float min_grad_ratio = ratios[0];
|
||||
for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]);
|
||||
float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, WEIGHT_FLOOR_FRACTION * min_grad_ratio);
|
||||
scratch[scratch_weight_floor] = adaptive_floor;
|
||||
|
||||
// Per-component weights — blend pre-floor, then enforce floor, then RENORM.
|
||||
// Renormalization is critical: PopArt's reward-magnitude EMA assumes a
|
||||
// consistent reward-scale; sum(weights) drifting > 1 would inflate the
|
||||
// running |reward| and contaminate the popart target every fold.
|
||||
float blends[6];
|
||||
float blend_sum = 0.0f;
|
||||
for (int c = 0; c < 6; c++) {
|
||||
float ratio = isv[REWARD_COMPONENT_GRAD_RATIO_BASE + c];
|
||||
float winner_weight = ratio; // exploit
|
||||
float diversifier_weight = (1.0f - ratio) / 5.0f; // explore
|
||||
float blend = improving * winner_weight + stagnant_or_worse * diversifier_weight;
|
||||
float weight = fmaxf(floor, fminf(1.0f, blend));
|
||||
scratch[scratch_weight_base + c] = weight;
|
||||
float winner_weight = ratios[c]; // exploit
|
||||
float diversifier_weight = (1.0f - ratios[c]) / 5.0f; // explore (sums to 1−ratios[c]·5/5 across c, see note)
|
||||
float w = improving * winner_weight + stagnant_or_worse * diversifier_weight;
|
||||
w = fmaxf(adaptive_floor, fminf(1.0f, w));
|
||||
blends[c] = w;
|
||||
blend_sum += w;
|
||||
}
|
||||
// Renormalize so Σweights = 1 exactly. EPS_DIV guards the (impossible)
|
||||
// all-floor-zero case.
|
||||
float inv_sum = 1.0f / fmaxf(blend_sum, EPS_DIV);
|
||||
for (int c = 0; c < 6; c++) {
|
||||
scratch[scratch_weight_base + c] = blends[c] * inv_sum;
|
||||
}
|
||||
|
||||
// Curiosity bound = fraction of recent PnL magnitude (signal-relative)
|
||||
float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA] * 0.3f; // 0.3 is Invariant-1 fraction
|
||||
float curiosity_bound = isv[PNL_REWARD_MAGNITUDE_EMA_INDEX] * CURIOSITY_BOUND_FRACTION;
|
||||
scratch[scratch_curiosity_bound] = curiosity_bound;
|
||||
|
||||
// Curiosity pressure: ALWAYS-ON exploration signal scaled by stagnation.
|
||||
@@ -141,35 +213,53 @@ scratch[scratch_curiosity_bound] = curiosity_bound;
|
||||
// When stagnating: rises toward curiosity_bound to break out of fixed point.
|
||||
// Permanent floor pattern per pearl_blend_formulas_must_have_permanent_floor —
|
||||
// max() form, not blend, so curiosity never zeros even when policy is winning.
|
||||
float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound; // 0.2 = Invariant-1 fraction
|
||||
float curiosity_floor = CURIOSITY_PERMANENT_FRACTION * curiosity_bound;
|
||||
float curiosity_dynamic = stagnant_or_worse * curiosity_bound;
|
||||
float curiosity_pressure = fmaxf(curiosity_dynamic, curiosity_floor);
|
||||
scratch[scratch_curiosity_pressure] = curiosity_pressure;
|
||||
|
||||
// Saboteur intensity: continuous mapping of z → [SABOTEUR_MIN, SABOTEUR_MAX]
|
||||
// adjusted by engagement rate (per pearl_engagement_rate_self_correction).
|
||||
// If saboteur is rarely shifting Q (engagement → 0), back off.
|
||||
// modulated by engagement rate (per pearl_engagement_rate_self_correction).
|
||||
// Compute the z-term inside the bound, modulate by engagement, and re-clamp
|
||||
// to [SABOTEUR_MIN, SABOTEUR_MAX]. Without the post-clamp the engagement
|
||||
// floor (0.1) would silently pull the output below SABOTEUR_MIN.
|
||||
float engagement = isv[SABOTEUR_ENGAGEMENT_RATE_INDEX];
|
||||
float intensity_z_term = SABOTEUR_MIN + (SABOTEUR_MAX - SABOTEUR_MIN) * improving;
|
||||
float saboteur_intensity = intensity_z_term * fmaxf(engagement, 0.1f); // 0.1 is engagement floor
|
||||
float modulated = intensity_z_term * fmaxf(engagement, ENGAGEMENT_FLOOR);
|
||||
float saboteur_intensity = fmaxf(SABOTEUR_MIN, fminf(SABOTEUR_MAX, modulated));
|
||||
scratch[scratch_saboteur_intensity] = saboteur_intensity;
|
||||
|
||||
// Per-component weight floor: signal-driven from worst-suppressed component's
|
||||
// grad-EMA. Idea: if any component has been consistently below mean grad,
|
||||
// its floor rises so it gets a chance to contribute.
|
||||
float min_grad_ratio = ratios[0];
|
||||
for (int c = 1; c < 6; c++) min_grad_ratio = fminf(min_grad_ratio, ratios[c]);
|
||||
float adaptive_floor = fmaxf(WEIGHT_HARD_FLOOR, 0.5f * min_grad_ratio); // 0.5 = Invariant-1 fraction
|
||||
scratch[scratch_weight_floor] = adaptive_floor;
|
||||
```
|
||||
|
||||
The constants `0.3f` (curiosity fraction of PnL), `0.5f` (floor fraction
|
||||
of min grad ratio), `0.2f` (CURIOSITY_PERMANENT_FRACTION — exploration
|
||||
floor as fraction of curiosity_bound), and `0.1f` (engagement floor) are
|
||||
Invariant-1 fractions: they bound rates, not regimes, and remain valid
|
||||
across all regimes. The thresholds themselves (what counts as "improving",
|
||||
"stagnant", "regressing") are entirely encoded in the Z-score and sigmoid
|
||||
— no hardcoded threshold survives.
|
||||
#### 3.4.1 Output smoothing (Pearls A+D applied to controller outputs)
|
||||
|
||||
The 10 controller outputs are written to `scratch_*` first, then a chained
|
||||
`apply_pearls_ad_kernel` launch smooths each output into its ISV slot.
|
||||
This prevents the controller from oscillating outputs at every training
|
||||
step (raw `improving` is responsive but jittery). Pearl A handles the
|
||||
fold-boundary cold-start; Pearl D's Wiener-α adapts smoothing strength to
|
||||
each output's own noise level (curiosity may be smoother than weights, etc.)
|
||||
|
||||
This means the consumer kernels read the **smoothed** controller outputs
|
||||
from ISV, not the raw scratch values.
|
||||
|
||||
#### 3.4.2 Surviving constants (Invariant-1 anchors only)
|
||||
|
||||
All seven hardcoded constants in the controller bound *rates*, not *regimes*,
|
||||
and remain valid across regime shifts:
|
||||
|
||||
| Const | Value | Role |
|
||||
|---|---|---|
|
||||
| `EPS_DIV` | `1e-6f` | divide-by-zero protection |
|
||||
| `WEIGHT_HARD_FLOOR` | `0.01f` | absolute weight floor — prevents zero |
|
||||
| `SABOTEUR_MIN` / `MAX` | `0.5f` / `2.0f` | curriculum rate limiter |
|
||||
| `CURIOSITY_PERMANENT_FRACTION` | `0.2f` | exploration floor as fraction of `curiosity_bound` |
|
||||
| `CURIOSITY_BOUND_FRACTION` | `0.3f` | curiosity ceiling as fraction of PnL EMA |
|
||||
| `WEIGHT_FLOOR_FRACTION` | `0.5f` | adaptive floor as fraction of min grad ratio |
|
||||
| `ENGAGEMENT_FLOOR` | `0.1f` | saboteur engagement floor (pre-bound) |
|
||||
|
||||
The thresholds that classify regime ("improving / stagnant / regressing")
|
||||
are entirely encoded in the Z-score and sigmoid — no hardcoded threshold
|
||||
survives at the regime layer.
|
||||
|
||||
### 3.5 Consumer migrations
|
||||
|
||||
@@ -179,12 +269,46 @@ across all regimes. The thresholds themselves (what counts as "improving",
|
||||
- `c51_loss_kernel.cu:789` — same migration
|
||||
- Other reward shaping multipliers identified during implementation audit
|
||||
|
||||
**Curiosity bonus** (NEW reward dimension):
|
||||
- New term in reward computation: `r_total += isv[CURIOSITY_PRESSURE_INDEX] *
|
||||
novelty_signal(state, action)`
|
||||
- `novelty_signal` is the simplest workable form: state-action visit count
|
||||
EMA → 1 / sqrt(count) (RND-style optional later)
|
||||
- Bounded by `CURIOSITY_BOUND_INDEX` in the producer kernel
|
||||
**Curiosity bonus** (NEW reward dimension, applied at *loss-compute* time
|
||||
not at experience-collect time — see §3.5.1):
|
||||
- The replay buffer continues to store the **base reward** only (no
|
||||
curiosity baked in). Curiosity is added to `r` at the moment a tuple is
|
||||
drawn for replay, recomputed against the *current* visit-count table
|
||||
and *current* `CURIOSITY_PRESSURE_INDEX`.
|
||||
- Recompute formula: `r_with_curiosity = r_base + isv[CURIOSITY_PRESSURE_INDEX]
|
||||
* novelty_signal(state, action)`
|
||||
- Bounded above by `CURIOSITY_BOUND_INDEX` in the producer kernel.
|
||||
|
||||
#### 3.5.1 Why recompute-at-replay (not store-at-collect)
|
||||
|
||||
If we store curiosity-augmented r in replay, old tuples retain the
|
||||
curiosity bonus they had *at the moment they were collected*, even though
|
||||
the policy now visits those state-actions frequently (curiosity should be
|
||||
near zero). Replaying those tuples then teaches the policy to revisit the
|
||||
fixed point we're trying to escape — making the ep1-overfitting pathology
|
||||
worse, not better. Recomputing at replay binds curiosity to the *current*
|
||||
exploration deficit, which is the only signal that matters.
|
||||
|
||||
The cost is one extra hash-lookup + multiply per replayed tuple. Existing
|
||||
replay sampling kernel adds two ops; no extra kernel launch.
|
||||
|
||||
#### 3.5.2 Novelty signal (state-action visit count via locality-sensitive hash)
|
||||
|
||||
The state space is continuous 42-dim. Use a SimHash-style discretization:
|
||||
- Project state through a fixed (untrained) random matrix `W ∈ R^{42×16}`
|
||||
with sign-quantized output: `code = sign(state @ W)` → 16-bit hash.
|
||||
- Bucket = `(action_id, code)` — i.e., 108 actions × 65,536 codes = ~7M buckets.
|
||||
- Visit count = atomic-free per-bucket EMA in a hash table sized 1M slots
|
||||
(with linear-probe collisions tolerated; collisions = false novelty
|
||||
underestimates, which is the safe direction).
|
||||
- `novelty_signal = 1 / sqrt(1 + count)`. Range [0, 1].
|
||||
- Hash table is per-fold (cleared at fold boundary via state-reset registry).
|
||||
- The 42×16 projection matrix is fixed at trainer init from a deterministic
|
||||
seed (so different runs with same seed get same hash function).
|
||||
|
||||
This is a cheap, GPU-friendly stand-in for RND or count-based bonuses. RND
|
||||
or full state-action density estimation can replace it later if smoke
|
||||
shows the hash discretization is too coarse.
|
||||
|
||||
**Saboteur intensity** (modulator on existing saboteur scale):
|
||||
- `gpu_experience_collector.rs:3328` — `let ps =
|
||||
@@ -195,15 +319,31 @@ across all regimes. The thresholds themselves (what counts as "improving",
|
||||
|
||||
### 3.6 Reset semantics
|
||||
|
||||
All 15 new slots: FoldReset (sentinel 0, Pearl A bootstraps from first
|
||||
observation). Cross-fold persistence is wrong here because the
|
||||
"improvement signal" should reset at each fold's cold-start.
|
||||
All **20** new slots get FoldReset entries (sentinel 0, Pearl A bootstraps
|
||||
from first observation). Cross-fold persistence is wrong for every slot
|
||||
in this spec because the "improvement signal" must reset at each fold's
|
||||
cold-start — train history from fold N-1 is not informative about
|
||||
fold N's improvement trajectory.
|
||||
|
||||
**Exception:** `REWARD_COMPONENT_GRAD_RATIO_BASE` slots — these accumulate
|
||||
across the fold and reset to sentinel at fold boundary. The other slots
|
||||
(`val_sharpe_delta`, `saboteur_engagement_rate`) need at least 2 epochs
|
||||
within a fold to compute a delta; they're signal-bootstrapped so the first
|
||||
epoch of each fold uses the floor (no signal yet).
|
||||
This includes:
|
||||
- 6 component weights (340..346)
|
||||
- 4 controller scalar outputs (346..350)
|
||||
- 2 val-sharpe canaries (350..352)
|
||||
- 6 grad-ratio canaries (352..358)
|
||||
- 1 saboteur engagement (358)
|
||||
- 1 PnL magnitude EMA (359)
|
||||
|
||||
**Cold-start window:** The Z-score is undefined until at least 2 val
|
||||
emits exist within the fold (need delta + variance). For epoch 0 of each
|
||||
fold, the controller observes `delta_ema = 0` (post-Pearl-A bootstrap),
|
||||
which gives `dz = 0`, `improving = 0.5`, `stagnant_or_worse = 0.5` — the
|
||||
"perfectly stagnant" mid-state. Every controller output is well-defined
|
||||
in this state: weights blend 50/50, curiosity sits at midpoint, saboteur
|
||||
sits at the geometric mean of [MIN, MAX]. There is no NaN path.
|
||||
|
||||
The **novelty hash table** (§3.5.2) also resets at fold boundary; this is
|
||||
NOT an ISV slot but a separate device buffer cleared by the state-reset
|
||||
dispatcher.
|
||||
|
||||
## 4. ISV slot allocation
|
||||
|
||||
@@ -217,7 +357,7 @@ After SP10's last slot at 339:
|
||||
| `348` | `REWARD_WEIGHT_FLOOR_INDEX` (adaptive) | 1 |
|
||||
| `349` | `CURIOSITY_BOUND_INDEX` (signal-relative) | 1 |
|
||||
| `350` | `VAL_SHARPE_DELTA_EMA_INDEX` (canary) | 1 |
|
||||
| `351` | `VAL_SHARPE_STD_EMA_INDEX` (canary) | 1 |
|
||||
| `351` | `VAL_SHARPE_VAR_EMA_INDEX` (canary, true variance) | 1 |
|
||||
| `[352..358)` | `REWARD_COMPONENT_GRAD_RATIO_BASE` (6 canaries) | 6 |
|
||||
| `358` | `SABOTEUR_ENGAGEMENT_RATE_INDEX` (canary) | 1 |
|
||||
| `359` | `PNL_REWARD_MAGNITUDE_EMA_INDEX` (canary) | 1 |
|
||||
@@ -228,9 +368,10 @@ Total: **20 new slots**. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`.
|
||||
|
||||
1. `reward_subsystem_controller_kernel.cu` (the main controller, ~200 LOC)
|
||||
2. `val_sharpe_delta_compute_kernel.cu` (canary — reads val sharpe history,
|
||||
writes Δ + |Δ| EMAs, ~80 LOC)
|
||||
3. `saboteur_engagement_compute_kernel.cu` (canary — reads per-episode Q
|
||||
shift after saboteur params change, writes engagement rate, ~100 LOC)
|
||||
writes delta_ema + delta_var_ema (true variance), ~80 LOC)
|
||||
3. `saboteur_engagement_compute_kernel.cu` (canary — per-bar
|
||||
|reward_with_saboteur − reward_without_saboteur| > EPS_ENGAGEMENT
|
||||
reduced via block tree-reduce, EMA'd; see §3.3.1 ~100 LOC)
|
||||
4. `reward_component_grad_ratio_compute_kernel.cu` (canary — reads
|
||||
per-component grad EMA from existing slots, normalizes to ratios, ~60 LOC)
|
||||
|
||||
@@ -266,10 +407,9 @@ Plus modifications to existing kernels:
|
||||
curiosity always (overfitting prevention).
|
||||
- **Component starvation guard**: Per-component weight reweighting could in
|
||||
principle starve a contributor. Mitigated by adaptive floor
|
||||
`≥ 0.5 × min_grad_ratio` ensuring even the least-contributing component
|
||||
keeps a non-trivial budget.
|
||||
Mitigated by adaptive floor `≥ 0.5 × min_grad_ratio` ensuring even the
|
||||
least-contributing component keeps a non-trivial budget.
|
||||
`≥ 0.5 × min_grad_ratio` and post-floor renormalization to Σ=1, ensuring
|
||||
every component keeps a non-trivial budget without inflating total
|
||||
reward magnitude (which would contaminate PopArt).
|
||||
- **Saboteur intensity adapts both directions**: When val sharpe is
|
||||
improving, saboteur intensity rises to keep the policy under realistic
|
||||
adversarial load (you don't want a model that only works when execution
|
||||
@@ -286,53 +426,131 @@ Plus modifications to existing kernels:
|
||||
multipliers)? Current design uses a single scalar mult. Defer to SP11.1
|
||||
if needed after empirical results.
|
||||
|
||||
## 8. Implementation phases (atomic per `feedback_no_partial_refactor`)
|
||||
## 8. Implementation phases — full straight-up implementation
|
||||
|
||||
Single commit:
|
||||
Two layers, matching SP4/SP5/SP7. Layer A is additive infrastructure
|
||||
(no behavior change) split per producer for independent revertability;
|
||||
Layer B is the atomic consumer migration that flips the trainer onto
|
||||
the new controller. Smoke runs only at the end of each layer for
|
||||
validation — there is no gating preflight that pauses or defers work.
|
||||
|
||||
1. Allocate 20 new ISV slots in `sp5_isv_slots.rs` (range 340..360)
|
||||
2. Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`, layout fingerprint
|
||||
3. Add 4 new producer kernels (controller + 3 canaries)
|
||||
4. Register kernels in `build.rs`
|
||||
5. Rust launchers in `gpu_dqn_trainer.rs` for all 4 new kernels
|
||||
6. Wire launches into `training_loop.rs` at appropriate points
|
||||
7. **Audit + migrate ALL existing reward weight constants:**
|
||||
- `mse_loss_kernel.cu:318` (cf_weight = 0.3 → ISV)
|
||||
- `c51_loss_kernel.cu:789` (cf_weight = 0.3 → ISV)
|
||||
- Other shaping multipliers found during audit
|
||||
8. Add curiosity bonus term to reward path (NEW dimension)
|
||||
9. Modify `gpu_experience_collector.rs:3328` for saboteur_intensity multiplier
|
||||
10. State reset registry: 20 new FoldReset entries + dispatch arms
|
||||
11. Contract test passes
|
||||
12. New pearl: `pearl_reward_as_controlled_subsystem.md` authored
|
||||
13. MEMORY.md index entry added
|
||||
14. Audit doc: Fix 39 entry with full architecture summary
|
||||
15. HEALTH_DIAG line for `sp11_reward` showing all 10 outputs + improvement_z
|
||||
### Layer A — additive infrastructure (3 commits, no behavior change)
|
||||
|
||||
Total estimated: ~1300-1700 LOC across ~12 files. ~2.5-3 hours subagent work.
|
||||
**A0 — slot allocation + state reset registry (~150 LOC)**
|
||||
- Allocate 20 new ISV slots (340..360) in `sp5_isv_slots.rs`
|
||||
- Bump `SP5_SLOT_END = 360`, `ISV_TOTAL_DIM = 360`
|
||||
- 20 FoldReset entries + 20 dispatch arms in state-reset registry
|
||||
- Novelty hash table reset entry (1M-slot device buffer, fold-cleared)
|
||||
- Layout fingerprint bumped
|
||||
- Contract test passes (slot count + reset arms match)
|
||||
|
||||
**A1 — three canary producer kernels (~250 LOC)**
|
||||
- `val_sharpe_delta_compute_kernel.cu` (Δ + variance EMA, two-pass)
|
||||
- `saboteur_engagement_compute_kernel.cu` (per-bar engagement, Pearl C)
|
||||
- `reward_component_grad_ratio_compute_kernel.cu` (existing
|
||||
per-component grad EMAs → normalized ratios)
|
||||
- All Pearl-A bootstrapped, Pearl-D smoothed
|
||||
- `build.rs` cubin registration; Rust launchers in `gpu_dqn_trainer.rs`
|
||||
- Wire launches into `training_loop.rs`
|
||||
|
||||
**A2 — controller producer kernel (~250 LOC)**
|
||||
- `reward_subsystem_controller_kernel.cu` reads 5 canary slots,
|
||||
writes 10 outputs to scratch, chained `apply_pearls_ad_kernel`
|
||||
smooths scratch into ISV
|
||||
- Novelty hash table device buffer (1M slots × f32) + 42×16 SimHash
|
||||
projection matrix (deterministic seed, init at trainer construct)
|
||||
- Build.rs entry, launcher, training_loop wire-up
|
||||
- HEALTH_DIAG `sp11_reward` line emits 10 outputs + `improvement_z`
|
||||
|
||||
### Layer B — atomic consumer migration (1 commit, ~750 LOC)
|
||||
|
||||
`feedback_no_partial_refactor`: every consumer of the migrated
|
||||
contract migrates in one commit. The trainer is on the new controller
|
||||
end of this commit.
|
||||
|
||||
- `mse_loss_kernel.cu:318` — `cf_weight = isv[REWARD_CF_WEIGHT_INDEX]`
|
||||
- `c51_loss_kernel.cu:789` — same migration
|
||||
- Full audit + migrate every hardcoded reward-shaping constant in
|
||||
`cuda_pipeline/*.cu` (expected ~3-5 additional sites; implementer
|
||||
enumerates and migrates all of them)
|
||||
- `gpu_experience_collector.rs:3328` — `let ps = saboteur_perturbation_scale
|
||||
* isv[SABOTEUR_INTENSITY_MULT_INDEX]`
|
||||
- Replay sampling kernel: curiosity bonus added **at replay time**
|
||||
(§3.5.1) reading `isv[CURIOSITY_PRESSURE_INDEX]` × `novelty_signal`
|
||||
- Reward composition reads all 6 component weights from ISV
|
||||
- Stale doc-string sweep across migrated sites
|
||||
|
||||
### Layer C — validation + close-out
|
||||
|
||||
**C1** — L40S 5-epoch smoke (`multi_fold_convergence`) on Layer B HEAD
|
||||
**C2** — full T10 (3-seed × 3-fold × 50-epoch) on Layer B HEAD
|
||||
**C3** — audit doc: Fix 39 entry with architecture summary + canary
|
||||
evolution evidence + Layer B before/after metrics
|
||||
**C4** — `pearl_reward_as_controlled_subsystem.md` authored
|
||||
**C5** — MEMORY.md index entry
|
||||
|
||||
**Total estimated:** Layer A ≈ 650 LOC + Layer B ≈ 750 LOC + Layer C
|
||||
≈ 150 LOC = ~1550 LOC across ~12 files. Layer A: ~1.5 hours. Layer B:
|
||||
~1 hour. Layer C: ~30 min. Plus validation wall-clock (smoke ≈ 25 min,
|
||||
T10 ≈ 4 hr).
|
||||
|
||||
## 9. Validation criteria
|
||||
|
||||
After SP11 lands:
|
||||
### 9.1 Smoke (5-epoch L40S `multi_fold_convergence`)
|
||||
|
||||
**Smoke (5-epoch L40S):**
|
||||
- HEALTH_DIAG `sp11_reward` line emits with all weights in [floor, 1.0]
|
||||
- `improvement_z` evolves across epochs (not stuck at 0)
|
||||
- `curiosity_pressure` oscillates with stagnation
|
||||
- No regression in Fix 33-38 metrics (IQN warnings still 0, dir_entropy
|
||||
still > 0.5, val_active_frac still > 0.20)
|
||||
Hard-pass criteria (all must hold):
|
||||
- HEALTH_DIAG `sp11_reward` line emits every epoch with all weights in
|
||||
[adaptive_floor, 1.0] and Σweights ≈ 1.0 ± 1e-3 (renormalization works)
|
||||
- `improvement_z` is non-zero by ep2 (canary populates after Pearl A
|
||||
bootstrap on ep1's first delta)
|
||||
- `curiosity_pressure ≥ 0.2 × curiosity_bound` always (permanent floor)
|
||||
and `≤ curiosity_bound` always
|
||||
- `saboteur_intensity ∈ [0.5, 2.0]` always (post-clamp)
|
||||
- No regression in Fix 33-38 metrics: IQN parallel-branch warnings = 0,
|
||||
dir_entropy > 0.5, val_active_frac > 0.20
|
||||
|
||||
**T10 (50-epoch full validation):**
|
||||
- val sharpe trajectory shows continued IMPROVEMENT (not just stagnation
|
||||
or decline) at least into ep20
|
||||
- Q-value continues to grow with val sharpe (not decoupled like ep0-14
|
||||
baseline)
|
||||
- Component weight evolution visible in HEALTH_DIAG (different components
|
||||
win at different training stages)
|
||||
### 9.2 T10 (3-seed × 3-fold × 50-epoch L40S)
|
||||
|
||||
**Specific success metric:** `val_sharpe[ep20] > val_sharpe[ep1]`. The Fix
|
||||
33-38 baseline showed `ep1 > ep14` (peak at ep1, declining). SP11
|
||||
success = peak shifts to ep20+ or beyond.
|
||||
Aggregation: 9 trajectories. Per-trajectory peak-epoch is the epoch
|
||||
maximizing val sharpe within that trajectory.
|
||||
|
||||
Hard-pass criteria (all must hold):
|
||||
- **Median peak-epoch across 9 trajectories ≥ 10**
|
||||
(Fix 33-38 baseline: median peak-epoch = 1, where xkjkb seed-0 fold-0
|
||||
peaked at ep1.) This is the primary signal that the model continues
|
||||
learning past ep1.
|
||||
- **Median val sharpe at ep20 > 0.95 × median val sharpe at peak-epoch**
|
||||
(the controller may shift the peak forward; a 5% drop from peak by ep20
|
||||
is acceptable noise. A larger drop indicates regression.)
|
||||
- **Mean val sharpe across all 9 trajectories at ep20 ≥ Fix 33-38 baseline
|
||||
ep1 mean** (sanity check — SP11 cannot make ep20 *worse* than the
|
||||
baseline's best epoch.)
|
||||
|
||||
Soft criteria (informative, not gating):
|
||||
- Component weight evolution visible across training (different
|
||||
components dominate at different stages — std(weight[c]) > floor over
|
||||
full T10)
|
||||
- Q-value growth correlated with val sharpe (Pearson(train Q, val sharpe)
|
||||
> 0.3 within each trajectory; baseline showed Pearson = -0.7 at the
|
||||
ep0-14 window — the decoupled-overfitting signature)
|
||||
- Per-fold pattern: fold 0 may peak earliest, fold 2 latest (cumulative
|
||||
novelty hash effects), but no fold collapses (peak-epoch ≥ 5 in every
|
||||
fold of every seed)
|
||||
|
||||
### 9.3 If T10 fails
|
||||
|
||||
Fix-forward, not roll-back. Failure modes and responses:
|
||||
- Median peak-epoch < 10 but z-canary populated: controller has signal
|
||||
but isn't acting strongly enough — extend ISV outputs to additional
|
||||
shaping multipliers found during Layer B audit
|
||||
- z-canary stuck at 0: variance EMA is mis-tracking; review Pearl A
|
||||
bootstrap on `VAL_SHARPE_VAR_EMA`
|
||||
- Component weights collapse to floor: renormalization or floor
|
||||
computation bug; review §3.4 implementation
|
||||
- Curiosity dominates loss (`r_total / r_base > 1.0` median):
|
||||
CURIOSITY_BOUND_FRACTION too high; this is the only Invariant-1
|
||||
fraction we'd revisit and still not because we hardcoded a regime,
|
||||
but because the chosen rate-limiter was too permissive
|
||||
|
||||
## 10. New pearl: `pearl_reward_as_controlled_subsystem`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user