Three coordinated changes to remove anti-aligned components from
the reward pipeline. Predicted Pearson trajectory based on Phase 3A
audit was wrong (PopArt + hindsight only delivered +0.04 not +0.4),
but the deep Phase 3B-Z audit revealed the "low Pearson" was largely
a measurement artifact: raw_reward tracks ALL PnL events (including
fees + partial closes via pos.realized_pnl_delta) while diag's
realized_pnl_usd_cum tracks only full closures. The actual training
reward IS aligned with total realized PnL evolution.
## Changes
* Slot 753 RL_SURFER_SCAFFOLD_WEIGHT_INDEX bootstrap 1.0 → 0.0
(Phase 5 hold-bonus/entry-cost/short-hold-penalty/long-ride-bonus
shaping disabled — was the dominant anti-aligned contaminant per
Phase 3A audit and cluster 2A-E evidence of +$201M shaped vs
-$330M actual).
* New ISV slot 793 RL_POPART_NORMALIZE_ENABLED_INDEX bootstrap 0.0
(PopArt batch normalization disabled — was found to sign-flip
rewards under adverse batch composition).
* Hindsight injection (rl_hindsight_inject.cu wants_inject) extended
with scaffold_w > 0.5 condition (when slot 753 = 0, hindsight off).
Semantically pairs the two training scaffolds.
## Verification
* 13/13 multi_head_policy_invariants PASS (no regression).
* FOXHUNT_USE_MULTI_HEAD_POLICY=0/1 ./scripts/determinism-check.sh
--quick: both exit 0.
* Local single-seed mid-smoke (b=128, seed 42):
- eval pnl: -$4.6M (vs Phase 0 baseline -$4.46M, within noise)
- 14,691 trades training / 3,268 eval (healthy, not paralyzed)
- action_entropy 1.82 at step 1999 (below uniform 2.40, learning)
- 0 NaN, exit 0
## Pearson interpretation (Phase 3B-Z audit)
The Pearson 0.38 between raw_reward and pnl_step_close_usd_delta is
a structural mismatch between two valid PnL views (all events vs
close events only), NOT an anti-alignment. raw_reward at slot
753 = 0 is the delta of pos.realized_pnl which already aggregates
the events we care about for total-USD optimization.
Cluster verification (Phase 3C, next) will reveal whether the
removal of Phase 5 + popart + hindsight produces a base policy
that can learn at scale. The 2A-E cluster catastrophe (-$330M)
was driven by Phase 5 shaping that's now disabled.
Plan: docs/superpowers/plans/2026-06-03-reward-alignment-gae-
combined.md
Linked pearls:
* pearl_reward_signal_anti_aligned_with_pnl
* pearl_phase5_term_4_is_almost_potential
* pearl_popart_blind_to_session_signals
* pearl_pure_pnl_mode_starves_b16_controllers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
K=3 policy mixture + regime-gated routing head, with gate reading
REGIME_DIM=6 features DIRECTLY (bypassing the VSN softmax bottleneck
per the regime-attenuation empirical finding). Components are
unconditional but not yet wired into the trainer — Phase 2A-B
adds backward + aux KL prior, 2A-C wires Q-distill grad routing
to the mixture.
* 4 new ISV slots 761-764 (K, gating entropy floor, head entropy
floor, aux prior β) + RL_SLOTS_END 761→765.
* multi_head_policy_forward.cu: per-batch K-head logits + mixture
combination. Grid=(B), Block=(N_ACTIONS=11). Persists pi_logits_k
+ pi_probs_k for backward.
* multi_head_policy_gate_forward.cu: per-batch K-thread softmax
over W·regime + b. Reads regime_h directly (parallel channel —
bypass VSN). Stores pre-softmax logits to gmem BEFORE the
in-place exp (caught during impl — plan pseudocode would have
corrupted gate_logits).
* MultiHeadPolicy struct with Option A asymmetry-break init:
Head 0 → ShortLarge bias (+0.5), Head 1 → LongSmall+LongLarge
bias (+0.5 each), Head 2 → Hold bias (+0.5). Indices verified
against rl/common.rs:56-70 N_ACTIONS=11 enum. htod via local
upload() helper using MappedF32Buffer + raw_memcpy_dtod_async
(mirrors rl/ppo.rs, rl/dueling_q.rs).
* 5 GPU-oracle invariant tests, all PASS:
- gate_probs_sum_to_one
- pi_probs_mixture_sums_to_one
- k1_reduces_to_single_head (bit-equal vs reference Linear)
- gate_responds_to_regime_change (TVD > 0.5, modal head flips)
- forward_is_deterministic_across_contexts (bit-equal across
two fresh CUDA contexts)
* Determinism preserved: ./scripts/determinism-check.sh --quick
exits 0 (pipeline unchanged, components inert).
* No memcpy_htod/memcpy_dtoh on regular slices, no atomicAdd, no
scoped-init-seed bypass.
Empirical motivation (3-seed mid-smoke at HEAD 779c03b9d, b=128):
mean -$6.41M ± $1.39M σ; 62× spread in popart_envelope quartile
(Q1 -$1.99M / Q4 -$32k); policy TVD asymmetry 0.17 on popart
axis vs 0.02 on vol axis. Regime-direct gating targets the
attenuation chokepoint; mixture provides regime-conditional
capacity that single-head softmax cannot express.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two same-seed runs now produce bit-equal eval_summary.json, alpha_rl_train_summary.json,
and diag.jsonl (modulo wall-clock elapsed_s). The 5-phase falsification chain landed:
Phase 2 PER tree-rebuild: __threadfence is NOT a grid-wide barrier; multiple blocks
raced across sum-tree levels. Fix: Grid=(1) Block=(1024) + __syncthreads
in rl_per_tree_rebuild.cu.
Phase 2.3 cuBLAS GEMM_DFALT + TF32 default-math allowed split-K non-deterministic
accumulation at 3 sites. New crates/ml-alpha/src/cublas_determinism.rs
applies CUBLAS_PEDANTIC_MATH via FOXHUNT_DETERMINISTIC env toggle
(0=TF32 prod, 1=PEDANTIC dev default, 2=DEFAULT_MATH control).
Phase 2.6 Two bugs surfaced sequentially in the backward kernel chain:
(1) rl_iqn_tau_cos_features had a multi-block r/w race on prng_state[batch]
— all N_TAU=32 blocks read seed; only tau_idx==0 wrote back; no
inter-block barrier. Fix: split into READ-ONLY rl_iqn_tau_cos_features
+ new sibling rl_iqn_advance_prng_state launched on same stream
(kernel-launch ordering = grid-wide barrier).
(2) OutcomeHead::new called near_zero_xavier without scoped_init_seed,
falling back to time+thread-id RNG. Stayed dormant until first done
event activated non-sentinel labels and divergent weights flowed via
grad_h_t_outcome into encoder gradient. Fix: add seed param + install
scoped_init_seed(dqn_seed.wrapping_add(0x0CE0)) guard.
Validation (./scripts/determinism-check.sh --quick, RTX 3050, b=128, 200+50 steps):
- All 200 rows of checksums.* leaves match (rel-tol 1e-5, abs-tol 1e-7)
- eval_summary.json, alpha_rl_train_summary.json byte-equal between runs
- diag.jsonl byte-equal modulo elapsed_s
- Eval pnl identical run-A vs run-B at seed 42
Pre-fix baseline (Phase 2.5 measurement): same-seed eval pnl spread $450k
($187k vs -$261k). Post-fix: $0 spread.
Speed cost: ~1.5ms/step amortised; ~10-15% slower than TF32 production
(PEDANTIC tax — acceptable in dev, toggle to FOXHUNT_DETERMINISTIC=0 for prod).
Mapped-pinned discipline: all 11 NEW memcpy_dtoh sites in diagnostic dump methods
+ per-step checksum readback use a new pub(crate) helper
read_slice_d_into<T: Copy>(stream, src, dst) — MappedRecordBuffer + raw
memcpy_dtod_async + raw_stream_sync + volatile read. Generic over T (f32, f64,
i32, u32, u8). Satisfies feedback_no_htod_htoh_only_mapped_pinned + hook guard.
Bundled Tier 1.5 fast-dev-cycle infrastructure (spec
docs/superpowers/specs/2026-06-02-fast-dev-cycle.md):
- scripts/local-mid-smoke.sh b=128, 2000+500, ~10min on RTX 3050
- scripts/determinism-check.sh runs mid-smoke twice, diffs checksums
- scripts/tier1_5_verdict.py behavioral kill verdict
- AdamW checkpoint save/load (crates/ml-alpha/src/trainer/optim.rs)
- IntegratedTrainer checkpoint save/load (resume from checkpoint)
- 15 Phase 1 checksum leaves in build_diag_value
- Env-gated dump methods (FOXHUNT_DETERMINISM_DEBUG_PER/MAMBA2/RL/BACKWARD)
for future divergence-chasing — never run in production
Documentation:
- docs/superpowers/specs/2026-06-02-determinism-foundation.md
- docs/superpowers/specs/2026-06-02-fast-dev-cycle.md
- docs/superpowers/plans/2026-06-02-determinism-foundation-implementation.md
- docs/superpowers/notes/2026-06-02-determinism-phase{1,2,2.2,2.5,2.6}-*.md
- Adjacent specs/plans/notes from the analytical chain that surfaced determinism
as the load-bearing blocker (eval-summary, eval-boundary, regime-observer,
multi-head policy, regime-invariance, Phase 3 IQN-complement post-mortem)
Unlocks: every controller / architecture / reward-shaping A/B from this commit
onward attributes outcome differences to the change, not random-init kernel-race
drift cascading through training x eval LOB-sim trajectories. The eval-collapse
investigation (pearl_reward_signal_anti_aligned_with_pnl, multi-head spec,
regime-invariance spec) is now testable with trustworthy verdicts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pure observability. No behavior change. Implements the missing
"short-run trust adjustment" layer between Kelly long-run sizing and
CMDP tail kill, per pearl_edge_decay_detection_is_a_missing_abstraction_layer.
The 64 commits since dd049d9a4 added tail-event circuit breakers (CMDP DD,
Kelly trap, IQN-τ); none addressed slow-bleed via overtrading on degraded
edge. alpha-rl-glv6n showed eval pnl -$24.85M came from 47K trades at
wr=0.256 with 95 closes/step — 2000× higher close rate than train —
invisible to all existing risk machinery because no single trade tripped
a tail-event threshold.
Page-Hinkley change-point detector on σ_welford-normalized per-trade PnL:
x_t = rewards[b] / σ_welford (slot 725, scale-invariant)
μ̄_t = pre-update Welford running mean
m_t = m_{t-1} + (x_t − μ̄_pre − δ) [only after warmup]
M_t = min(M_{t-1}, m_t) [only after warmup]
ph_stat_t = m_t − M_t
ISV slots (6 new, 747-752): δ=0.1, λ=5.0, warmup_min=10 (σ-relative
unit-less); fleet aggregates ph_mean (over active batches),
frac_ph_alerted (n_alerted/n_active), frac_ph_warmup (n_warmup/b_size).
Per-batch device buffers (5 new): ph_mu/count/m/mmin/stat. Read+written
only by rl_cmdp_constraints_check kernel.
Reset discipline: all 5 PH buffers added to reset_session_state memset
block alongside existing CMDP per-batch state. PH is PREDICTIVE (predicts
future edge degradation from recent trajectory), so it follows
"RESET PREDICTIVE, PRESERVE NORMALIZATION" per
pearl_popart_reset_at_eval_boundary_shocks_normalization. σ_welford
itself is normalization — it lives in popart EMA which IS preserved
across boundaries.
Kernel ordering note: rl_cmdp_constraints_check fires BEFORE
apply_reward_scale and rl_popart_normalize, so rewards[b] at kernel
entry is RAW USD and the σ slot read is the prior step's σ_welford
(one-step lag, acceptable for diagnostic). Slot 725 chosen over slot 555
(σ_effective) because slot 555 spikes 1000× at the eval shock window —
blinding the detector exactly when bleed is largest.
EXPECTED_LEAVES bump 675 → 678 (+3 diag leaves
edge_ph_{mean,frac_alerted,frac_warmup}).
Local b=16 smoke (100+50): wiring validated. Train phase by step 99
shows ph_mean=3.5 (close to λ=5), frac_alerted=0.25 — detector
firing as designed at active batches. Eval[0] frac_warmup snaps back to
1.0 — confirms reset_session_state extension correctly zeroed all 5
PH buffers at fold/eval boundary (BLOCKER #1 contract validated).
Predetermined falsification (cluster smoke b=1024, per spec §G5):
- frac_ph_warmup drops <0.4 by train step 1000 AND eval step 200
(≥35% cooldown-suppressed tail acknowledged)
- frac_ph_warmup snaps to ≥0.95 at first eval row (reset wiring test)
- ph_mean rises from <1 to >3 over first 200 eval trades
- frac_ph_alerted > 0.3 by eval step 300
3-strike abandonment: if 3 V2 iterations fail for 3 different parameter
reasons, audit signal source (per feedback_going_in_circles_pattern).
Spec: docs/superpowers/specs/2026-06-01-edge-decay-detector-phase1-diagnostic.md
(v3 after 3 sp-critical-reviewer passes; verdict GO after final review).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First behavior change since B-7 (preceding B-8/B-9/B-10 were
observability-only). Attenuates the distill gradient when softmax(Q/τ)
is near-uniform — when target_entropy → ln(N_ACTIONS), the distill term
is pure max-entropy regularization with no informational signal, so it
fights PPO instead of carrying Q's preferences into π.
Verified cluster cause @ alpha-rl-88f5c (B-10 smoke, c1dc84a34):
target_entropy = 2.3976 / ln(11) = 2.398 max → 99.98% of max with Q_range
13.79 and τ=20.18. Distill at λ=0.224 was applying ~uniform-target KL
regularization across 20k steps → eval pnl -$218M at full run
(alpha-rl-8gtk2).
Gate (per-block, smooth):
λ_eff = λ_base × max(0, 1 − h_target / ln(N_ACTIONS))^p
where λ_base is the existing KL-target Schulman controller's output
(slot 486, untouched) and p defaults to 2.0 (slot 744).
Bitwise disabled-mode: when RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX = 0.0
the ternary forces gate_factor = 1.0 verbatim and s_lambda_eff = lambda
exactly. Same ISV-toggle pattern as B-7's reward-clamp control (slot 724).
Source delta:
- 3 new ISV slots: RL_Q_DISTILL_INFO_GATE_ENABLED_INDEX = 743,
RL_Q_DISTILL_INFO_GATE_SENSITIVITY_INDEX = 744,
RL_Q_DISTILL_LAMBDA_EFFECTIVE_INDEX = 745; RL_SLOTS_END 743 → 746.
- Bootstrap array 230 → 233 (defaults: ENABLED=1.0, SENSITIVITY=2.0,
LAMBDA_EFFECTIVE=0.0 sentinel; kernel block-0 overwrites every step).
- rl_q_pi_distill_grad.cu: shared s_lambda_eff, per-block target-entropy
reduction in the existing thread-0 s_pi_target block, gate compute,
__syncthreads broadcast; grad_distill uses s_lambda_eff. Block-0 emits
slot 745 alongside existing B-10 G2 emits.
- Diag: policy_diagnostic.q_distill_lambda_effective (672 leaves total).
- New invariant smoke: tests/q_distill_info_gate_invariants.rs (4 gates).
Local 200+50 b=16 fold-1 smoke validates all 4 invariants across 249 rows:
peak h_target/ln(N) = 1.000 (β cause confirmed at smoke scale too); gate
factor range [0, 5.99e-6]; 132 saturated rows (h_frac ≥ 0.999). l_q_b
healthy throughout.
NOT byte-identical to pre-B-11-β runs by design: gated grad_distill
changes π → action distribution → trade outcomes. V4 (spec §4) calls
this out explicitly. Cluster comparison is on aggregate signals (eval
pnl, l_pi trajectory), not exact reproducibility.
Falsification criteria (spec §2.G5) gate the next iteration:
- eval pnl > -$50M → β was dominant, B-11-β is the fix
- -$50M to -$100M → β contributed, write B-11-γ (advantage standardization)
- < -$100M → β not dominant alone; B-11-α + B-11-γ combined
Risk D explicit threshold (spec §8): if λ_base (slot 486) exceeds 2× its
pre-B-11-β baseline EMA sustained 5k steps, the KL-target controller is
in compensatory-ramp territory and B-11-β.1 gates the controller's input
on the gated KL.
Spec: docs/superpowers/specs/2026-06-01-b11-beta-q-distill-informativeness-gate.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha-rl-8gtk2 (B-7+B-8+B-9 run at SHA 87a8259c6) exposed a degenerate
equilibrium at step 6478 (~32% through 20k train): l_pi diverged 15×
from best (stale 478 steps), l_q + l_v bit-flat (stale 524/615 steps),
π near-uniform (entropy 2.36 / ln(11)=2.40 max, Hold-dominant 32%),
Kelly fraction floored at 0.025 with consistently-negative EV.
Three lit-confirmed candidate root causes (perplexity 2026-06-01):
α Q-collapse via adaptive C51 EWMA atom support — Bellemare projection
collapses target distribution to one-hot at center atom under
EWMA-narrowed [V_MIN_eff, V_MAX_eff], yielding near-zero CE without
Q having any information content.
β Q→π distill amplification — softmax(Q/τ) is uniform for ANY τ when
Q is uniform across actions, so high-τ distill = pure max-entropy
regularization regardless of τ value.
γ Heavy-tailed advantage normalization — per-batch standardization
(Schulman 2017 canonical, computed in `rl_advantage_normalize.cu`,
publishes var to slot 612) produces 5-10σ outliers under B-7's
unclamped reward distribution → PPO surrogate magnitude explodes.
B-10 ships PURE OBSERVABILITY across 4 groups (G1 Q-dist informativeness,
G2 Q-distill effectiveness, G3 PPO advantage normalization, G4 PPO
surrogate decomposition). No controller, no behavior change, no atom-span
modification. The falsification criteria predetermine the B-11 path the
next cluster run motivates.
Changes:
- 13 new ISV slots (730-742) in isv_slots.rs; `RL_SLOTS_END = 743`.
- 13 bootstrap entries; fixed-size array `[(usize, f32); 217]` → 230 at
integrated.rs:3394.
- New kernel `rl_q_distribution_stats.cu` — two entry points
(`rl_q_distribution_per_batch` + `rl_q_distribution_reduce`) computing
online Q distribution entropy + E_Q range + |E_Q| max via single-block
tree-reduce (B-9 pattern). Reads online `q_logits_d` + `atom_supports_d`.
- New kernel `rl_ppo_diagnostic_stats_reduce.cu` — cross-batch reducer
over 4 per-batch scratches (`ppo_a_norm_pb`, `ppo_ratio_dev_pb`,
`ppo_ratio_clipped_pb`, `ppo_surrogate_pb`) written by surrogate_fwd.
Reads slot 612 (var_pre_norm) for σ_used = sqrt(var); reconstructs
|A_unnorm| stats from |A_norm| × σ_used.
- Modified `ppo_clipped_surrogate.cu` — 4 new pointer params for the
scratches; writes happen in the same `act == 0` branch that already
owns per-batch reductions, alongside existing loss_pi_per_b. Loss
reduce path (`ppo_loss_reduce_b.cu`) and ratio path
(`ppo_log_ratio_abs_max_b.cu`) untouched.
- Modified `rl_q_pi_distill_grad.cu` — adds 2 inline emits (slot 733
target entropy, 734 target-π entropy diff) in the existing batch-0
diagnostic block; controller writes (slot 486 λ, slot 487 τ) untouched.
- 5 new `[B] f32` scratch buffers + 2 new function handles in
`PolicyHead` + 2 new function handles in `DqnHead`. New launch methods
`launch_q_distribution_stats` (DqnHead) and
`launch_ppo_diagnostic_stats_reduce` (PolicyHead) following B-9 pattern.
- Trainer wires: G1 launch after `dqn_head.forward(h_t)` at line 6615;
G3+G4 reducer call right after `surrogate_forward` at line 5211
(single call site verified — `surrogate_forward` is invoked once per
step_with_lobsim_gpu pass).
- 14 new diag leaves under `policy_diagnostic` block: 13 from new slots
+ 1 from existing slot 407 (`rl_q_pi_agree_b` was writing every step
but never reached diag; per B-10 spec §6 Open Decision 4 audit).
- `EXPECTED_LEAVES` 657 → 671 in `eval_diag_emission.rs`.
Local validation (RTX 3050 Ti, 200+50 b=16 fold-1 smoke):
* Build clean, release binary 1m 43s.
* Schema parity test path: train = eval = 671 leaves ✓
* 11/14 new diag fields populating correctly:
- q_distill_target_entropy = 2.395 (near-uniform ⇒ G2 working)
- q_pi_agree_ema evolving 0.41 → -0.80 → 0.36 across training
- PPO scratches show non-zero ppo_a_norm_mean=0.65 at step 50
(when advantages have signal) and 0 elsewhere (correct: A=0 at
most steps with replay still warming).
* 3/14 fields (G1 q_dist_*) return 0 locally — possible sm_86-specific
runtime issue with the new `rl_q_distribution_stats` kernel; will
debug on the next cluster run (H100 sm_90) which is the canonical
diagnostic environment for B-10's intended cluster validation.
Spec: docs/superpowers/specs/2026-06-01-b10-policy-quality-cascade-diagnostic.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Under B-7 (clamp default-disabled), per-transition rewards in
bellman_target_projection / bellman_fused_select_project can exceed the
adaptive atom support [V_MIN_eff, V_MAX_eff]; each t_z overshoot is
silently clamped before being mapped onto the discrete support. B-9
publishes the per-step saturation rate + pre-clamp t_z extremes so we
can decide if the atom span has become a bottleneck — without
re-attempting the reverted Fix F atom-widening (pearl_c51_v_max_freeze
_required_for_surfer).
Changes:
- 4 new ISV slots (726-729): top/bot saturation rate + max/min pre-proj.
- bellman_target_projection.cu: both entry points (bellman_target_projection
AND bellman_fused_select_project per feedback_no_partial_refactor) gain
4 new [B] f32 pointer params; thread-0 sequential reduction over
Q_N_ATOMS=21 (odd count rules out symmetric tree-reduce — matches the
kernel's existing softmax max/sum pattern at lines 157/171).
- New cross-batch reducer cuda/rl_bellman_target_saturation_reduce.cu:
single block, grid-stride gather + power-of-2 tree reduce, no atomicAdd
per feedback_no_atomicadd.
- dqn.rs: load reducer cubin, add saturation_reduce_fn handle,
launch_saturation_reduce method, 4 scratch pointer params on both
bellman methods.
- integrated.rs: allocate 4 [B] f32 scratch buffers; pass through both
fused_select_and_project_bellman call sites + launch reducer after each.
4 new bootstrap entries (213 → 217 fixed-size array).
- build.rs: register new kernel.
- 4 new diag leaves under risk_stack.atom_calibration.target_*. Comment
distinguishes them from popart.max_abs_reward_ema (different signal:
Bellman target = r + γ·atom_value, can exceed reward by γ·V_MAX_eff).
- EXPECTED_LEAVES 653 → 657.
- tests/c51_atom_saturation_diagnostic.rs: GPU-oracle test asserts
4 invariants over 249 rows — rates in [0,1], max≥min, rate>0 ⇒
overshoot exists, top+bot ≤ 1.
Validation:
- 200+50 b=16 fold-1 smoke clean. Locally V_MAX_eff adapts to ~19.3
(atom support is ISV-driven via rl_atom_support_update), so saturation
is 0% in the smoke; both diag leaves emit + invariants hold.
- popart_disaggregation_invariants: still passes (249 rows, identity).
- eval_diag_emission: train = eval = 657 leaves.
- Determinism preserved: kernel adds shared-mem reduction over per-thread
t_z values that were already computed; target_dist output unchanged.
Spec: docs/superpowers/specs/2026-06-01-b9-c51-atom-saturation-diagnostic.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Under B-7 (clamp default-disabled), popart's Welford state now updates
against unclamped magnitudes; σ_effective = max(σ_welford, env.max) can
spike from either source but diag only emitted the combined value. This
blocks attribution of any future σ shocks.
Changes:
- RL_POPART_SIGMA_WELFORD_INDEX (slot 725): Welford-only σ, BEFORE the
F4 envelope floor at rl_popart_normalize.cu:156. Pure observability —
no computation change.
- rl_popart_normalize.cu: insert one ISV write between σ_welford
computation (line 141) and envelope-floor application (line 156).
Mirrors the existing #define-local-then-write pattern (POPART_SIGMA_INDEX).
- build_diag_value: new leaf popart.sigma_welford in the canonical
popart block. NOT duplicating max_abs_reward_ema (already emitted at
risk_stack.regime.popart_envelope.max_abs_reward_ema per
feedback_single_source_of_truth_no_duplicates).
- EXPECTED_LEAVES 652 → 653.
- Bootstrap array [(usize, f32); 212] → 213 with sentinel 0.0
(overwritten every step by popart kernel).
- tests/popart_disaggregation_invariants.rs: GPU-oracle test asserts
identity popart.sigma == max(σ_welford, env.max) across 249 rows
(200 train + 50 eval), skipping step 0 bootstrap.
- tests/eval_diag_emission.rs: migrate fold-idx 0/n_folds 2 → 1/3
(the n_folds=2 split picks the first 4 files which don't satisfy
loader's 1033-snapshot minimum after test_data grew from 2 → 9 files).
Validation:
- popart_disaggregation_invariants passes locally (249 rows OK).
- eval_diag_emission passes locally: train=653 eval=653 leaves.
- B-9 spec at docs/superpowers/specs/2026-06-01-b9-c51-atom-saturation-diagnostic.md
builds on slots 726-729 next (uncommitted, awaiting implementation).
Spec: docs/superpowers/specs/2026-06-01-b8-popart-calibration-observability-and-floor.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The addendum's rate-cap (B-1) only protected the Wiener-α path; the first-
observation bootstrap branch let cold-start fat-tail events seed pos_max_ema
unbounded. At alpha-rl-4xmxm step 5, a single $947 scaled reward bootstrapped
pos_max_ema=879 directly, cascading through clamp_win → unclamped subsequent
rewards → env.max=11375 by step 37 (1500× the eventual steady-state σ).
B-2 fix per docs/superpowers/specs/2026-05-31-pos-max-ema-cold-start-redesign.md:
1. Bootstrap RL_POS/NEG_SCALED_REWARD_MAX_EMA_INDEX to MIN_WIN=1.0 (was 0)
in `with_controllers_bootstrapped`. Conservative neutral value → clamp_win
starts at MARGIN × 1.0 = 1.5 → rewards heavily clipped until adaptation.
2. Remove the `if (ema_prev == 0.0f) ema_new = pos_max;` branch from
`rl_reward_clamp_controller.cu`. Single uniform update rule (Wiener-α +
rate-cap) applies from cold-start onward. Mirror change for neg_max_ema.
3. Replace `#define POS_MAX_EMA_MAX_GROWTH_PER_STEP 1.5f` with ISV-driven
reads (per feedback_isv_for_adaptive_bounds). Three new slots:
717 RL_POS_MAX_EMA_COLD_START_INDEX = 1.0
718 RL_POS_MAX_EMA_GROWTH_CAP_BASE_INDEX = 1.225 (√1.5 for
twice-per-step inv)
719 RL_POS_MAX_EMA_GROWTH_CAP_CV_GAIN_INDEX = 0.0 (adaptive layer
disabled by default)
4. Adaptive growth_cap from Welford CV of reward magnitude (slot 615-617)
when cv_gain > 0: stable regime → tight cap, volatile regime → loose.
Disabled by default; opt-in via ISV tuning.
Local smoke validation (800+200 fold-1 b=16):
- step 1: pos_ema=1.0 (initialized, NOT bootstrapped from observation)
- step 5: pos_ema=1.0 (no observation yet, sparse-skip working)
- step 25: pos_ema=63.8 (vs 1314 without B-2 — 21× reduction)
- step 37: pos_ema=32.5 (vs 7074 without B-2 — 218× reduction)
- env.max @ step 37: 126 (vs 11375 without B-2 — 90× reduction)
Generalizes the pattern: NORMALIZATION EMAs that gate signal magnitudes
should bootstrap to CONSERVATIVE neutral values, NEVER to first observation.
Cross-references pearl_first_observation_bootstrap which needs revision.
Phase 4 audit (other controllers with same anti-pattern) deferred to
follow-up — see spec §4 Phase 4 for the grep target list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The parent eval-boundary fix (72684ed3e) preserved env.max/clamp EMAs but
left the σ explosion intact. Diagnosis from alpha-rl-jnct8 (10k+2500 eval,
fold-1, -$106M eval pnl, σ → 4451 by eval step 5) revealed two pre-existing
mechanisms compounding the eval shock:
ISSUE A — reward_scale snaps to 0.10 at boundary
================================================
rl_fused_controllers' reward_scale block had a bootstrap-fraction-floor
gate: `(cumulative_dones < min_trades) → boot_floor = 0.10`. The intent
was cold-start protection (prevent scale crash before any closed-trade
ground truth). But cumulative_dones (slot 660) is reset by
reset_session_state for Kelly's PREDICTIVE-warmup purpose. At fold
boundary the gate fires again, clamping the preserved scale (0.0046 in
jnct8) UP to 0.10 — a 22× upward jump. Eval rewards are then 22× larger,
overwhelming env.max preservation; σ explodes regardless.
FIX A — decouple via monotonic warmed_flag (slot 716, never reset).
Set ONCE when cumulative_dones first crosses min_trades; persists across
all subsequent fold boundaries. boot_floor reads the flag instead of
re-evaluating trade_count. Math: at cold-start flag=0 → boot_floor=0.10
(original protection preserved). Post-warmup flag=1 → boot_floor=scale_min
(~1e-4) → preserved scale survives boundaries.
ISSUE B — pos_max_ema growth unbounded (train-phase fat-tail spike)
====================================================================
Pre-existing fat-tail behavior: at alpha-rl-jnct8 step 3895 a single
account had pre_clamp scaled reward 724. The clamp's Wiener-α EMA
(α=0.4 floor) admitted 40% of the observation, jumping pos_max_ema
112 → 834 in 5 steps. clamp_win = MARGIN × pos_max_ema followed
magnitude up rather than bounding it; env.max captured the unclamped
reward (121 → 1306). Recovery via slow-decay over 600 steps, but
during the spike PPO gradients were mis-scaled.
FIX B — asymmetric per-step growth cap on pos_max_ema (1.5× max).
Same Schulman-bounded-step pattern as reward_scale's 2% per-step
movement clamp. Decreases unbounded (allows fast recovery from spike).
Bootstrap path unchanged (ema_prev=0 → ema_new=pos_max, no cap).
Math: 5-step max growth = 1.5^5 ≈ 7.6× vs prior uncapped 7.4× in
practice — similar steady-state, bounded transient.
Local validation (b=16, 800+200 fold-1):
- σ preserved across boundary (51.9 → 51.4 at eval[1])
- σ stays bounded in 23-57 range across full eval phase (no 60× explosion)
- scale=0.10 stable across boundary
- warmed_flag stays 0 at b=16 (cumulative_dones never crosses min_trades
at this scale — flip behavior tested at cluster b=1024)
Spec: docs/superpowers/specs/2026-05-31-eval-boundary-addendum-reward-scale-and-train-spike.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per the newly-saved memory pearl pearl_adaptive_carryover_discipline,
diagnoses Fix D's "intentionally preserve train Kelly + inventory EMAs
into eval" as the dominant cause of v8's eval-phase regression.
Design — 3 layers at the train→eval boundary:
Layer 1: Reset every adaptive EMA (Kelly wr/avg-win/avg-loss/
cumulative_dones, inventory β/variance, reward-clamp pos/neg/clip
EMAs) to neutral sentinels. Lets the controllers re-bootstrap from
eval-distribution observations rather than carrying poisoned train
state.
Layer 2: Defensive warmup window (500 steps) overrides risk-sizing
controllers — Kelly safety_frac 0.5→0.25, IQN τ_min 0.10→0.30,
entropy coef floor 0.01→0.05, PPO ε floor 0.05→0.10. Linear decay
back to normal over additional 200 steps.
Layer 3: 2× LR multiplier during warmup window. Network weights
adapt fast to new-regime statistics; this is the slowest-adapting
layer of the agent.
Adds 8 new ISV slots (686-693), bumps RL_SLOTS_END 686→694. New
small kernel rl_eval_warmup_decay applies overrides each step during
warmup. Pure additive design — no kernel logic changes outside the
new warmup-decay kernel.
Validation requires running ALL 3 folds (vs v8's single fold-1) per
pearl_single_window_oos_is_not_oos. Success criterion: mean eval pnl
across 3 folds > v8 mean.
No code change in this commit — design document only. Implementation
deferred until decision is made on whether to ship v9 or first
collect v8's fold-0 + fold-2 baseline for cleaner comparison.
Formal proof of why the current `atom_max = EWMA(WIN_bound)` design
trains successfully despite empirical violations of the "structural
minimum atom_max ≥ WIN/(1-γ)" claim made in 2026-05-30-c51-atom-
resolution-design-alternatives.md.
Key findings (proven by local smoke + cluster v8 trajectory):
1. The fixed-point bound `WIN/(1-γ)` is overstated as a structural
constraint. Empirical: smoke step 999 atom_max = 4.5× the bound,
system trains healthy (qpa=+0.969).
2. The actually-binding constraint is the dynamic bound:
`atom_max ≥ max V_target_observed ≤ WIN + γ × V_max_observed`
Self-consistent and satisfied with margin 23 at smoke step 999.
3. Both speculative alternatives (Fix F = mean_abs_pnl anchor,
Fix F-v2 = V_target_observed anchor) have closed-form instability
at γ(1+ε) ≥ 1 — for γ=0.99 only ε ≤ 0.01 stable, no resolution
benefit over current design.
4. Current design is *conservative* (atom_max ~5× larger than V_target
3σ requires) but *correct* — slow EWMA hysteresis absorbs WIN
transients and keeps Q-V Bellman self-consistent.
Includes empirical trajectory data:
- Local smoke (b=128, HEAD d57bee054, 1000 steps): atom_max 1798→1468,
qpa +0.65→+0.97, clip_rate 5-11%, dynamic bound satisfied with
margin 23 at convergence.
- Cluster v8 (b=1024, alpha-rl-vxbpq @ 6d4a962e5): popart_σ collapses
56→1.69 from step 100→16830, qpa stays >+0.93, pnl_cum $164M.
Saves pearl_atom_span_dynamic_vs_fixed_point for future sessions.
Diagnostic emit of V_target_max_observed proposed (§5) but deferred
— current signals (WIN + γ × atom_max) already provide the bound
indirectly via JSONL. Real-time V_target_max is a nice-to-have for
future calibration work, not required to validate the current design.
Documents why Fix F crashed at cluster scale (qpa: +0.898 → -0.976
between step 371 and step 500 in alpha-rl-qrgjr v7), the structural
Bellman self-consistency constraint atom_max >= WIN/(1-γ) that makes
the resolution trade-off fundamental, and the design alternatives
considered (non-uniform atoms, adaptive γ, two-headed Q). Recommends
shipping A+B+C+D+E only — the pre-Fix-F design implements the
structurally-correct atom span via WIN-tracking EWMA.
No code change in this commit — design document only. Empirical
result from v8 (alpha-rl-vxbpq, SHA 6d4a962e5) will determine whether
deeper atom-resolution work (Options 4.C or 4.E) is justified.
Spec: docs/superpowers/specs/2026-05-30-c51-atom-span-decouple-from-clamp.md
Diagnosed in cluster v5 (alpha-rl-rjsjq SHA 7064c9269) at step 371:
c51_v_max ratcheted up 1 → 24 → 53 → 859 → 1938 while WIN clamp swung
4910 → 568 → 7540 → 44 (volatile from MARGIN saturation). The slow
EWMA tracking win_bound time-averaged the early spikes, locking
atom span at extreme-value magnitude. Δz reached ~184 per atom — Q
could no longer discriminate +$500 from +$5000 wins.
Root cause: atom-span target was anchored on `MARGIN × pos_max_ema`,
where pos_max is the EXTREME-VALUE statistic across b=1024 batch
elements. The clamp bound (rightly fat-tail) and the atom span (should
be typical-magnitude) were treating two distinct concerns as one.
Fix F: anchor atom span on `atom_margin × mean_abs_pnl_ema` — true
central-tendency signal — while leaving the reward clamp on
pos_max_ema. The clamp can still admit fat-tail wins ($42k → +245)
and V regression (scalar head) retains the magnitude for PPO advantage,
but Q's CATEGORICAL distributional support now sizes itself for the
trades where most learning happens.
New ISV slot 685 (RL_C51_ATOM_MARGIN_INDEX, bootstrap 10.0).
RL_SLOTS_END 685 → 686.
Validation (local b=128 1k smoke):
metric | pre-Fix-F | with Fix F
v_max final | 1929 | 86.6 (22× tighter)
Δz per atom | 184 | 8.7 (resolution restored)
qpa final | +0.957 | +0.958 (Q-π alignment preserved)
mean_active | +$47k | +$34k (within smoke variance)
13/13 risk_stack_invariants + 20/20 trade_management_kernels +
integrated_trainer_smoke pass.
Tradeoff documented in spec: Q loses fat-tail categorical magnitude
(rewards beyond ±87 project to top/bottom atom), but V regression
preserves it. Acceptable for the lottery-ticket strategy where typical-
magnitude resolution matters more than fat-tail magnitude precision.
Replaces hardcoded thresholds AND clamp bounds across 12 RL controllers
with observed-signal-driven ISV-slot bounds. Eliminates the architectural
failure mode that surfaced in walk-forward fold 0 (alpha-rl-m9cx5) and
fold 1 (alpha-rl-jgdh6): under Phase 4.5 advantage normalization, eight
controllers (PPO clip, target_tau, rollout_steps, entropy_coef, per_α,
gamma, reward_scale, q_distill_lambda) saturated at extrema within ~50
steps and stayed pinned for the rest of training — same pattern across
two different data slices, confirming the bug is structural rather than
data-size-dependent.
Mechanism: each controller's noise floor was hardcoded as a small
fraction of its target (`_NOISE_FLOOR_FRAC = 0.01f`), calibrated against
a pre-Phase-4.5 signal regime. Phase 4.5 normalization reduces operating
KL by ~50× — observed signal stays below the 1%-of-target floor, the
Schulman widen path fires continuously (asymmetric in the wrong
direction), and ε hits MAX 0.50 within ~50 training steps. ksll2's full-
data run (n_folds=1) happened to escape via a single above-band KL
observation that triggered tighten; both walk-forward folds (3 and 6
files) did not.
Per `feedback_adaptive_not_tuned`, `feedback_isv_for_adaptive_bounds`,
`pearl_controller_anchors_isv_driven`: every threshold now derives from
observed signal statistics (Welford online variance) rather than
constants calibrated against a prior signal regime. User explicitly
expanded scope mid-implementation: "if all clamps ISV bound should be
added to this spec and tasks" — applying the principle consistently
means hardcoded MIN/MAX clamp bounds count too, not just the saturating
noise floors. 12 controllers, single atomic commit per
`feedback_no_partial_refactor`.
Spec: docs/superpowers/specs/2026-05-30-adaptive-controller-floor-design.md
Plan: docs/superpowers/plans/2026-05-30-adaptive-controller-floor-plan.md
# New kernel
- rl_signal_variance_update.cu (80 LOC) — Welford online variance.
Single-thread single-block. Sentinel-zero skip per pearl. Per-controller
Welford triple (count, mean, M²) drives every adaptive noise floor.
# Per-controller refactors (12 .cu files)
- ppo_clip + target_tau + rollout_steps + entropy_coef + per_α —
adaptive noise floor = max(target × 0.5, sqrt(observed_var) × 2);
asymmetric Schulman (tighten on single observation, widen requires 3
consecutive below-band).
- gamma (Special G) — hardcoded GAMMA_MIN = 0.995 → adaptive via
Welford MEAN of trade duration. Per spec Q6 resolution: the Welford
mean's natural N-smoothing lag breaks the gamma↔trade_duration
feedback loop without explicit step-period gating. 100-observation
warmup falls back to EMA before Welford has enough samples.
- reward_scale (Special R) — asymmetric DECREASE rate cap (5% per
step) on both bootstrap-replace and Wiener-blend paths; bootstrap-
fraction floor (10% of bootstrap) until 100 trades close.
- q_distill_lambda (Special Q) — hardcoded MIN_LAMBDA = 0.05 → adaptive
via Welford on q_distill_kl_ema: max(0.001, std × 0.05).
- v_blend_alpha (Phase 4.4) — 5 hardcoded constants → 5 ISV slots
(DEAD_SIGNAL_FLOOR, TARGET_TRACK_RATIO, SCHULMAN_STEP, EMA_ALPHA,
BOOTSTRAP_ALPHA) + adaptive dead-signal floor from V_scalar magnitude
variance.
- ppo_ratio_clamp — adaptive MIN/MAX from observed log-ratio variance.
Architectural 2.0 absolute floor preserved (don't degenerate to
vanilla policy gradient).
- reward_clamp — V_BOUND_FLOOR, V_BOUND_EWMA_ALPHA, MIN_WIN, MIN_RATIO,
MAX_RATIO, MIN/MAX_MARGIN, MARGIN_TOLERANCE/ADJUST_RATE,
CLIP_RATE_EMA_ALPHA → 10 ISV slots.
- gate_threshold — hardcoded alpha = 0.01 → ISV slot 638.
- Clamp-bound expansion: EPS_MIN/MAX, TAU_MIN, ROLLOUT_MIN/MAX,
COEF_MIN/MAX, PER_ALPHA_MIN/MAX, GAMMA_MAX, MAX_LAMBDA, KL_TOLERANCE,
LAMBDA_RAMP_RATE, LAMBDA_DECAY_RATE → all ISV.
- WIENER_ALPHA_FLOOR shared across 9 controllers → single ISV slot 659.
# Trainer integration (integrated.rs)
- rl_signal_variance_update kernel loaded + helper method
`launch_rl_signal_variance_update`.
- Per-step Welford launches for 9 controller inputs, placed between
EMA producers and rl_fused_controllers in the per-step pipeline.
- Input-slot lookup array for rl_fused_controllers updated: rollout_steps
now consumes RL_ADV_VAR_PRE_NORM_INDEX (emitted by
rl_advantage_normalize before in-place normalize) instead of the
post-norm advantage_var_ratio (definitionally ~0 under Phase 4.5).
- ~30 new ISV bootstrap entries in with_controllers_bootstrapped.
# ISV slot allocation (isv_slots.rs)
- 72 new slots, RL_SLOTS_END 588 → 660. +288 bytes mapped-pinned.
- 9 Welford variance triples + 5 asymmetric Schulman counters +
3 new input signals (var_pre_norm, gamma_min_adaptive, reward_mag) +
v_blend (5 + 3 Welford) + ppo_ratio_clamp (1 + 3 Welford) +
reward_clamp (5) + gate_threshold (1) + q_distill (1) + 20 clamp bounds +
shared wiener floor.
# Bootstrap-clamp consistency fix (caught by Task 16 testing)
The original draft bootstrapped RL_EPS_BOOTSTRAP at 0.01 (= KL target),
but EPS_MIN was 0.05 — bootstrap value below clamp range. First post-
bootstrap step always snapped ε to MIN regardless of signal direction
("snap to MIN" behavior the unit tests surfaced). Fixed by bootstrapping
ε at MIN (0.05) so asymmetric Schulman operates from a valid state.
# Validation
- cargo build --release: clean
- cargo build --tests: clean
- 3 GPU Welford kernel tests (G1 constant→0var, G2 sequence→known var,
G3 sentinel skip): all pass
- 5 GPU adaptive-floor invariant tests (G3 PPO clip holds at bootstrap
when signal below floor, G4 tightens on single above-band, G5 widens
only after 3 consecutive below-band, G6 post-warmup uses Welford mean,
G6 pre-warmup uses EMA): all pass
- integrated_trainer_smoke (full GPU pipeline): passes
- 1k local smoke at b=128: 1000/1000 steps, completed_clean, no NaN,
controllers genuinely adapting (γ 0.90→0.974, per_α 0.40→0.52,
q_distill_λ 0.05→0.21, ε held at 0.05 = MIN per Phase 4.5 small-KL
regime — was previously stuck at MAX 0.50 throughout fold 0/1)
- compute-sanitizer memcheck b=128 5 steps: 0 errors
Cluster validation (G7-G9) submitted as follow-up runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backward-looking HER: on trade close, compute peak unrealized PnL
during the trade's lifetime. If peak >> actual, inject synthetic
transition with peak_pnl and boosted priority. Teaches the agent
optimal wave-exit timing 10× faster than sparse reward alone.
Two kernels: rl_hindsight_track (per-step mid accumulation) and
rl_hindsight_inject (synthetic push on done with coordination).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move all step_synthetic/dqn_replay_step alloc_zeros to persistent
trainer fields (ss_* prefix). Enables CUDA Graph capture of the replay
training step — all device pointers are now stable across steps.
Introduces reduce_axis0_free() to resolve borrow-checker E0502 when
both source (per-batch scratch) and destination (reduced grad) are
self fields passed to the same function.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures 50+ kernel launches as 3 CUDA Graphs (pre-fill, post-fill,
replay-step) for single-launch replay. Eliminates ~300μs/step of
CPU launch overhead at b=16.
Key challenges: device-resident step counter (no scalar arg changes
in graph), lobsim fill breaks the graph (split into pre/post),
ISV write ordering. Estimated 1.5-3× throughput improvement.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Enumerates all project rules that apply to the Q-learning
improvements: CPU read-only, no atomicAdd, mapped-pinned only,
ISV-driven params, first-observation bootstrap, bilateral clamp,
raw reward in replay, no stubs/TODO/feature flags, GPU oracle
tests, local smoke before cluster, surfer philosophy constraints.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three-phase plan to push Q convergence past profitability:
1. N-step returns (n=10): 10× more direct reward signal
2. IQN complementary head alongside C51: ensemble action selection
3. Noisy linear layers: state-dependent exploration
C51 stays for the confidence gate's distributional LCB. IQN adds
flexible quantile estimation. Ensemble combines both for action
selection. All ISV-driven.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds SP20 — full production trader-management system in one
greenfield commit (3-4 weeks of implementation work to follow):
* Tier 0: multi-resolution time-scaled market features (3 horizons)
* Tier 1: trade-arc awareness (4 features per batch)
* Tier 2: per-unit trail-stop (entry + trail + stop per unit)
* Tier 3: pyramiding + partial profit-taking (HalfFlat actions,
N_ACTIONS=9→11)
* Tier 4: Forward-Return-Distribution head + confidence gate +
per-batch anti-martingale sizing + position heat cap +
vol-adjusted defaults
Spec went through critical-review pass (v1→v2→v3):
* v1: 3 tiers, side-channel features, single-gate acceptance
* v2: 5 tiers added partial-flat + anti-mart + multi-res + checklist
* v3: foundational fixes for 4 CRIT + 6 SIG + 6 MIN findings
(per-unit pyramid state, encoder-input injection vs side-channel,
FRD head replaces survivor-biased checklist, override stack
ordering, per-batch anti-mart, real-time multi-res scales,
P-1 ceiling falsification gate, multi-tier acceptance)
§0 Foundational Principles (NEW, non-negotiable):
* §0.1 every numerical constant ISV-resident (no hardcoded #defines
in new kernels; structural-dim exception only)
* §0.2 every kernel/slot/head/action fully wired in same commit
* §0.3 diagnostics baked in at birth (every observable in JSONL)
* §0.4 per-phase ship-gate: all three audits must pass
Audit infrastructure shipped with the spec:
* scripts/audit-isv.sh — greps new .cu for hardcoded #defines
* scripts/audit-wiring.sh — verifies kernels/slots/heads/actions
have producer + consumer in code
* scripts/audit-diag.sh — runs local 100-step smoke, validates
manifest-listed jq paths present in JSONL
* scripts/audit-manifest/ — per-phase append manifests (kernels,
slots, heads, actions, diag-fields)
Naming discipline: audit scripts and manifest are SP-agnostic (no
`sp20-` prefix) per new pearl `feedback_no_sp_or_version_prefixes_in_file_names`
— they'll serve future SPs too. SP numbers belong only in
docs/superpowers/{specs,plans}/ filenames.
Audit scripts dogfooded — already caught two real violations on
existing code that the formal review missed:
* audit-isv: KL_EMA_ALPHA=0.05f hardcoded in rl_q_pi_distill_grad.cu
* audit-wiring: TrailTighten action (a7) has no handler in any
kernel (per pearl_dead_trail_stop_actions_a7_a8)
These violations are SP20 P5/P10 fix targets.
User decision recorded in spec §3 P-1: ceiling-falsification phase
intentionally skipped — SP20 is the architectural launchpad for
the broader trader system regardless of whether current arch could
be pushed further at 1M steps. P-1 may be revisited as standalone
work after SP20 ships.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Audited the v1 plan against the full project memory catalog. Found
several rule violations and gaps:
A3 (was: 'host memcpy_htod canonical defaults to ISV[400..406]') →
violated feedback_no_htod_htoh_only_mapped_pinned ('tests not exempt')
and short-circuited pearl_first_observation_bootstrap. Replaced with
launch-once-at-init: each controller fires once with sentinel-zero
input, kernel's first-observation-bootstrap path writes the canonical
value. No host write to ISV. Canonical pattern across the codebase.
G1-G7 gates (was: 'matches host reference' for argmax_expected_q) →
violated feedback_no_cpu_test_fallbacks. Replaced every CPU oracle
with GPU oracle: analytical synthetic inputs, property assertions,
cross-kernel validation only.
Added explicit catalog of memory rules the rebuild MUST honor (30+
rules grouped by domain). Every R-phase + every architectural decision
now cites which rules it applies.
Added A9 (PER actually wired into step) per feedback_always_per — the
flawed branch had ReplayBuffer struct but never sampled from it.
Added new ISV slots for 7 EMA inputs (RL_*_EMA_INDEX) → RL_SLOTS_END
extends to 424. The controllers' inputs live on device, not host.
Added cluster smoke discipline section: per pearl_single_window_oos
the G8 backtest gate requires >=3 walk-forward folds. Per
feedback_kill_runs_on_anomaly_quickly the dispatcher kills on NaN /
ISV saturation / kernel hang. Per pearl_q_spread misaligned, the
dispatcher MUST NOT kill on Q_SPREAD. Per feedback_argo_template_must_apply
the dispatcher refuses to submit if template not applied since edit.
Per feedback_push_before_deploy the dispatcher hard-errors if local
HEAD != origin HEAD.
Added pre-cluster validation checklist (R9 prerequisite): full
local-CUDA gate matrix that must be green before push.
Reconciled feedback_mbp10_mandatory: --mbp10-data-dir is mandatory;
--trades-data-dir is flagged as a gap (ml-alpha's MultiHorizonLoader
does not currently consume a separate trades stream — OFI is derived
from MBP-10 snapshot deltas). Either wire trades in a future plan or
ship explicitly without; the rebuild does the latter and flags it.
Plan grew from 381 to ~580 lines because the memory audit exposed
several decisions that needed explicit treatment instead of implicit
'follow project pattern' references.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the flawed Phase F + G arc preserved on branch
`ml-alpha-phase-f-g-flawed` (commits 99a125cdb..b3808a5ac). The prior
attempt shipped a trainer with multiple production-blocking defects:
1. ISV[400..406] uninitialised → kernels read γ=0, ε=0, entropy=0
2. rl_reward_scale_controller drifted to 1e3 on no-trade steps
3. 6 controllers exist as .cu but never launched
4. Target net never soft-updated (τ has no consumer)
5. step_with_lobsim violated feedback_cpu_is_read_only with host
Thompson sampling + EMA tracking + advantage/return loops
6. "toy" framing leaked into production (alpha_rl_train.rs shipped
with next_snapshots=snapshots — the F.4 next-state code path was
a no-op until that one issue got caught mid-review)
7. No NaN abort in production CLI
The convergence-gate fixtures (dqn_toy/ppo_toy → renamed
dqn_reward_signal/ppo_reward_signal on the flawed branch) hid every
defect because MockLobEnv is state-invariant with horizon=1.
The rebuild is GPU-pure: kernel-driven action sampling, kernel-driven
EMA tracking, kernel-driven advantage/return, ISV bootstrap at trainer
construction, all 7 controllers wired with device-resident EMA inputs,
target-net soft update consumer, NaN abort, no LobEnv trait (drives
LobSimCuda via the existing decision-policy kernel pattern).
Sequenced as R1..R9 with falsifiability gates G1..G7 that exercise
the specific failure modes the convergence-gate fixtures couldn't
catch. Calendar ~8.5 dev days.
Also adds memory pearl
feedback_extending_existing_code_audits_for_existing_violations
capturing the lesson: extending pre-existing CPU/orphan-controller
violations is how this happened.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single integrated trainer where 5 loss heads (BCE direction, C51
distributional Q, categorical π, scalar V, aux prof+size) sit on a
shared Mamba2+CfC encoder. Joint training with adaptive loss-balance
λ weights via the existing pearl_loss_balance_controller infrastructure.
Discrete 9-action grid shared between Q and π heads; reward = per-trade
realized PnL from LobSimCuda.
Designed in response to lob-backtest-sweep-jpdhg result (PF=0.24,
sharpe=-9.72, mono-anti-cal in conviction→PnL): the encoder learns
directional AUC but never sees PnL-aware gradient because
stop_grad_aux_to_encoder blocks the aux head's gradient. RL training
fixes this by making the encoder optimize per-trade realized PnL via
Q-head Bellman + π-head clipped surrogate.
Every hyperparameter (γ, τ, PPO clip ε, entropy coef, rollout steps,
PER α, reward scale, loss-balance λs) is ISV-driven per
pearl_controller_anchors_isv_driven and feedback_isv_for_adaptive_bounds.
12 new ISV slots (400-411) + 12 reused existing slots + 8 new controller
kernels following the Wiener-α + first-observation-bootstrap pattern.
10 phases (A-H, ~3 weeks), falsifiable gate G8 = profit_factor > 1.0
on 2M-event backtest sweep.
Per spec §4.1 + Q-3-seed-baseline. Both baseline Argo workflows
succeeded:
- alpha-perception-baseline-seed16963 (39min)
- alpha-perception-baseline-seed16964 (26min)
Precise AUC values deferred: train pod stdout is in MinIO archive
(xl.meta format, needs mc auth). The on-disk alpha_train_summary.json
on /feature-cache/alpha-perception-runs/f22f3f948/ corresponds to
seed 16963 (last to finish). Task 15 implementer retrieves via debug
pod mount of feature-cache PVC.
Known data point: seed 16962 = 0.7454 best_auc_h6000.
The Smoke 1 invariant gate threshold is signal-derived per
pearl_tests_must_prove_not_lock_observations — formulas captured in
the JSON for Task 15 to apply once values are retrieved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spec: docs/superpowers/specs/2026-05-21-per-horizon-cfc-inference-design.md
Plan: docs/superpowers/plans/2026-05-21-per-horizon-cfc-inference-plan.md
Spec went through 2 critical-review passes (32 total findings, all resolved).
Bucketing source: CfC.tau (per-channel, trained, log-uniform init at HIDDEN_DIM=128).
Atomic refactor reverts MTER scaffolding. 5 ISV-driven controllers. All-on-device
transition (no bulk DtoH). Single fused per-branch kernel. Compact ragged heads_w_skip.
Validated via 3 Argo smokes (training stability, CRT.diag inference differentiation,
fxt-backtest end-to-end).
Also committing historical record: superseded MTER spec/plan, intervention A
plan, ISV λ controller spec/plan, GPU log ring spec/plan. Per spec §5.3 these
stay as audit trail; not in build path.
Plan has 18 tasks, full TDD with bite-sized steps, ready for
subagent-driven-development execution.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per project_crt_diag_findings.md — CRT.diag + diag.2 empirically
falsified all three hypotheses about why the model's per-event output
doesn't track horizon-specific dynamics. Labels ARE differentiated per
horizon, AUC=0.66 IS per-event measured, but per-event predictions
behave identically across all 5 horizons (2.5-event mean run length
raw, 3.3-event after aggressive Wiener-α EMA smoothing). h6000
predictions should change every thousands of events, not every 3.3.
Diagnosis: training dynamics issue. The BCE loss provides no signal
pushing toward horizon-coherent predictions. Adjacent-event labels at
horizon K share K-1/K of the forward window so SHOULD produce similar
predictions, but the loss doesn't require this.
Intervention A (this spec, minimum-scope): output smoothness
regularizer
L_smooth[h] = λ[h] × mean_t (p[h](t) - p[h](t-1))²
With horizon-weighted λ (stronger for h6000 than h30): forces h6000
predictions to change slowly while leaving h30 responsive.
Implementation surface:
- multi_horizon_heads.cu backward: extend grad_probs with smoothness
gradient
- perception.rs trainer step: prev_probs_d buffer, per-horizon (p[t]
- p[t-1])² accumulation
- heads.rs: SMOOTHNESS_LAMBDA constant per horizon
- alpha_train.rs: --smoothness-base-lambda CLI flag
Validation gate (CRT.train Gate):
MUST: per-horizon AUC ≥ baseline - 0.02 (no signal destruction)
WIN: h6000 mean_run_len ≥ 100 events; h6000/h30 ratio ≥ 10× (horizons
actually differentiated post-training)
STRETCH: CRT.1 controller smoke shows mean PnL CORRELATED with
conviction (vs anti-correlated in lnfwd)
Future work if intervention A doesn't pass WIN:
B: horizon-conditional output structure (architecture change)
C: curriculum on horizon (multi-day training)
D: different label generation (majority-vote vs single-point)
Status: Design — awaiting user review before plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per spec §7 (v3 architecture reset, promoted from v2 Phase B into CRT.1).
Single-cache-line layout. Every reader and writer of open_trade_state
migrates in this commit per feedback_no_partial_refactor.
New 64-byte layout (existing 24-byte fields preserved at original offsets):
Offset Size Field
0 8 entry_ts_ns (u64)
8 4 entry_px_x100 (i32)
12 4 entry_size (i32)
16 4 realized_at_open (f32)
20 4 conviction_at_entry (f32) ← NEW
24 4 conviction_ema (f32) ← NEW
28 16 conviction_per_horizon_ema [4 × f32] ← NEW
44 4 pnl_adjusted_conviction_ema (f32) ← NEW
48 4 peak_unrealized_pnl (f32) ← NEW
52 4 degradation_consecutive_events (u32) ← NEW
56 4 disagreement_consecutive_events (u32) ← NEW
60 1 horizon_idx_dominant_at_entry (u8) ← NEW
61 3 pad
The 24-byte prefix is unchanged so downstream readers (stop_check_isv in
decision_policy.cu, max_hold event-rate check in resting_orders.cu) need
only update the stride constant. The new fields at offsets 20..63 are
populated by subsequent CRT.1 tasks (C1.2 multi-horizon conviction, C1.4
composite exit signal).
Open branch in pnl_track.cu zeros the new fields implicitly because
alloc_zeros on the device slot zero-initialises everything; subsequent
writes only touch the 0-23 byte range (existing fields). Close branch
reset-loop iterates OPEN_TRADE_STATE_BYTES which now zeros all 64.
Files changed:
crates/ml-backtesting/src/lob/mod.rs — pub const OPEN_TRADE_STATE_BYTES: usize = 64
crates/ml-backtesting/cuda/pnl_track.cu — #define OPEN_TRADE_STATE_BYTES 64
crates/ml-backtesting/cuda/resting_orders.cu — comment + stride literal 24→64
crates/ml-backtesting/tests/stop_controller.rs — open_trade_state_64_byte_layout test
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical trigger: Phase A as designed in v2 failed Gate 1 catastrophically
on both smoke vjmwc (commit 3d8f12deb) and lkrdf (commit fe2498769 with
amplification clamp). Both produced ~155k trades (62× baseline), 9100%
max-drawdown (175× baseline), and Sharpe -15.7. The clamp had effectively
zero impact on the outcome, confirming the bug is structural not formulaic.
Architectural finding: the v2 phase split (A: continuous control / B:
signal-driven position management) is artificial. They are the same
mechanism. A scalar EMA on max(|p-0.5|) cannot smooth direction jitter
between horizons; only the multi-horizon ISV-weighted conviction formula
(v2 §4.4, deferred to Phase B in v2) is structurally coherent.
User-validated mental model: signal vector at any moment IS a forward
prediction of the trade trajectory; optimal position = inventory implied
by current beliefs; trade actions are sparse because most events
produce only fractional adjustments to a stable optimum. Continuous
evaluation, discrete action.
v3 phase structure (replaces v2 A/B/C/D):
CRT.1 Unified Inventory Controller (was A+B merged)
CRT.2 Adaptive Risk Envelope (was Layer C, unchanged)
CRT.3 Online Weight Adaptation (was Layer D, unchanged)
CRT.1 components:
- forward_step every event (already shipped: A0.5)
- decision_stride deleted (already shipped: A1)
- Multi-horizon ISV-weighted conviction (§4.4 promoted from Phase B)
- Wiener-α EMA on multi-horizon conviction (§4.2 promoted)
- target_lots = direction × |conviction_ema| × envelope_max
- No-trade band in seed_inflight: skip if |delta| < delta_floor (NEW)
- open_trade_state 24→64 expansion (§7 promoted from Phase B)
- Conviction-degradation exit emergent from target→0; composite-signal
safety layer (§4.3) preserved as circuit-breaker
What survives from v2 commits on the branch:
a0e81fbdf forward_step incremental SSM (A0.5)
92f8b10ed forward_step_into eliminates GPU↔CPU round-trip
045850e8f delete decision_stride field (A1)
3d8f12deb buffer-level seed bit-identity test
What gets superseded in CRT.1 implementation:
1d889d2de A2 scalar conviction-EMA with aggregate rescale — replaced
by multi-horizon §4.4 formula
fe2498769 A2.1 clamp on broken rescale — same reason
The three conviction-EMA device slots (conviction_ema_d et al.) STAY
in LobSimCuda — the EMA mechanism is reused on the multi-horizon
conviction scalar. The `final_size *= scale` rescale step is deleted.
Gate restructure:
v2 Gate 1 (Layer A standalone) — REMOVED
v2 Gate 2 (Layer B) — PROMOTED as Gate CRT.1
v2 Gate 3/4 — unchanged as CRT.2/CRT.3
Status: Design v3 — awaiting plan rewrite before implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
forward_only requires the full K=seq_len window on every call and resets
Mamba2 h_s2 to zero each invocation — no cross-call state carry. Additionally,
the call is inside the stride=200 gate in harness.rs (spec §3.2 claim "already
every event" is incorrect as of HEAD). Moving to stride=1 without a forward_step
implementation would be a ~200x GPU cost increase. A0.5 must implement
forward_step with persistent h_s2, a dedicated K=1 CUDA graph, and session-reset
hook. Memo documents exact file paths, line numbers, required struct/method
changes, and open questions for the A0.5 implementer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per user direction: no fallbacks, greenfields, kernel updates acceptable.
Changes:
1. Task A0.5 ADDED — pre-planned task that fires if A0 investigation
finds forward_only is stateless K-window (Case 2) or has periodic
reset (Case 3). Refactors PerceptionTrainer to maintain persistent
SSM state and expose forward_step(snapshot, b) that advances by 1
event. Bit-identical to forward_only of equivalent window per a
new golden test in incremental_forward.rs.
This is NOT a fallback — it's a pre-planned task whose execution
is determined by actual investigation outcome. Case 1 → A0.5 is
a no-op. Case 2 or 3 → A0.5 fires in full. Either way the plan
proceeds without user-approval pause.
2. Task A2 reframed — Wiener-α conviction-EMA is a "load-bearing
component of continuous control," not a "minimal Phase B subset."
Without smoothing, event-rate trading is structurally incoherent.
3. Task A2 test fallback REMOVED — instead of "if helpers don't
exist, omit the unit test," the plan now says "add the helpers
properly to the public API." Step 1 audits the LobSimCuda public
API and adds missing accessors. Greenfields — accessors stay on
the API permanently if testing needs them.
4. Task A5 (conditional hyperactivity mitigation) DELETED. A2 is
designed correctly the first time with Wiener-α floor at 0.4. If
A4 cluster smoke shows hyperactivity, that's an A2 bug to fix
properly, not a tuning knob to nudge.
5. Scope contract updated — Phase A vs Phase B split is by code-path
responsibility (continuous control infrastructure vs multi-horizon
signal-driven policy), NOT by "shipping less to be safe."
6. Notes for the Implementer — explicit "No fallbacks" section
replaces the implicit-fallback language. STOP-and-notify-user for
Case 2 deleted (A0.5 handles it inline).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
