705d6c156b4afc732da2ce26e89bc2caa4d41baf
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jgrusewski
|
705d6c156b |
audit: ISV-ify 10 more design constants — Schulman + bootstraps + streaming α
Per `feedback_isv_for_adaptive_bounds` + user "do all except floors
and clamp bounds": 10 more constants moved from kernel-side `#define`s
into ISV slots (78 slots total now).
## Slot additions (468-477)
RL_SCHULMAN_TOLERANCE_INDEX (468, =1.5) — shared by 4 controllers
RL_SCHULMAN_ADJUST_RATE_INDEX (469, =1.5) — shared by 4 controllers
RL_STREAM_ALPHA_INDEX (470, =0.05) — shared by var + kurt streaming
RL_KURT_GAUSSIAN_INDEX (471, =3.0)
RL_KURT_NOISE_FLOOR_INDEX (472, =1.0)
RL_TAU_BOOTSTRAP_INDEX (473, =0.005)
RL_EPS_BOOTSTRAP_INDEX (474, =0.2)
RL_ROLLOUT_BOOTSTRAP_INDEX (475, =2048)
RL_REWARD_SCALE_BOOTSTRAP_INDEX (476, =1.0)
RL_PPO_RATIO_CLAMP_BOOTSTRAP_INDEX (477, =10.0)
## Skipped (per user "do all except floors and clamp bounds")
* `*_MIN`/`*_MAX` clamp bounds (algebraic domain — risk γ=1.5 nonsense)
* Numerical floors: ABS_MEAN_FLOOR=1e-6, M2_SQ_FLOOR=1e-12, EPS_PNL=1e-3
(risk div-by-zero if mis-tuned)
* C51 atom layout (V_MIN/V_MAX) — architecture, not config
## Wiring
* Shared Schulman pattern: 4 controllers (ppo_clip, target_tau,
rollout_steps, plus per_α independent KURT slots) now read TOLERANCE
+ ADJUST_RATE from the same 2 ISV slots. Single source of truth.
* Each controller's bootstrap (1st-emit on sentinel-zero) reads
isv[*_BOOTSTRAP_INDEX] instead of #define value. The `prev ==
BOOTSTRAP` first-observation replace-direct check also reads from
ISV.
* 2 streaming kernels (var + kurt) share RL_STREAM_ALPHA_INDEX.
## Diag bake-in
JSONL `isv_config` block grows by 10 new fields: schulman_tolerance,
schulman_adjust_rate, stream_alpha, kurt_gaussian, kurt_noise_floor,
tau_bootstrap, eps_bootstrap, rollout_bootstrap,
reward_scale_bootstrap, ppo_ratio_clamp_bootstrap. Total isv_config
fields: 26.
Also includes windowed action_entropy fix (was structurally 0 at
b_size=1) — accumulates EMA-smoothed action distribution over
~1k-step window, computes entropy on the windowed dist. Makes the
exploration metric meaningful at b_size=1.
## Slot total
RL_SLOTS_END: 468 → 478. **78 total ISV slots.**
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
827a0e9416 |
fix(rl): ISV-ify ALL remaining tunable design constants (10 new slots)
Per `feedback_isv_for_adaptive_bounds`: every controller design knob
that's genuinely tunable now lives in ISV instead of as a kernel-side
`#define`. Tuning is a re-seed (kernel launch with new arg) rather
than a recompile.
## New ISV slots (10 design constants)
RL_REWARD_CLAMP_WIN_INDEX (452, =1.0) apply_reward_scale
RL_REWARD_CLAMP_LOSS_INDEX (453, =3.0) apply_reward_scale
RL_KL_TARGET_INDEX (454, =0.01) rl_ppo_clip_controller
RL_IMPROVEMENT_THRESHOLD_INDEX (455, =0.99) rl_lr_controller
RL_PLATEAU_PATIENCE_INDEX (456, =1000.0) rl_lr_controller
RL_DIV_TARGET_INDEX (457, =0.01) rl_target_tau_controller
RL_ENTROPY_TARGET_FRAC_INDEX (458, =0.7) rl_entropy_coef_controller
RL_KURT_LIFT_SCALE_INDEX (459, =7.0) rl_per_alpha_controller
RL_PPO_CLAMP_MARGIN_INDEX (460, =10.0) rl_ppo_ratio_clamp_controller
RL_LR_WARMUP_STEPS_INDEX (461, =2000.0) rl_lr_controller
RL_SLOTS_END: 452 → 462.
## Constants NOT converted (truly fundamental)
* All `*_INDEX` (ABI)
* All `*_MIN`/`*_MAX` clamp bounds (algebraic domain)
* All `*_BOOTSTRAP` (one-shot init)
* `WIENER_ALPHA_FLOOR` (per pearl_wiener_alpha_floor_for_nonstationary)
* Schulman pattern parameters (`*_TOLERANCE`/`*_ADJUST_RATE`)
* C51 (`Q_N_ATOMS`, `V_MIN/MAX`, `N_ACTIONS`)
* Kernel numerics (`STREAM_ALPHA`, `ABS_MEAN_FLOOR`, `EPS_PNL`)
* `KURT_GAUSSIAN` (statistical constant = 3.0 for Gaussian)
* `KURT_NOISE_FLOOR` (defensive)
* `LR_BOOTSTRAP`/`LR_MIN`/`LR_MAX`/`LR_LOSS_EMA_ALPHA`/`DECAY_FACTOR`
## New infrastructure
New CUDA kernel `rl_isv_write.cu` — generic single-thread device-side
seeder taking `(int slot, float value)`. Trainer loops calling it
once per design constant at init. Replaces the prior pattern of
extending `rl_streaming_clamp_init`'s arg list every time a new
constant was added.
## Ordering fix
Design constants must be seeded BEFORE controllers bootstrap — the
controllers' bootstrap paths read these slots (e.g.
`rl_entropy_coef_controller` reads `RL_ENTROPY_TARGET_FRAC_INDEX`
to derive its target). Without correct ordering, controllers see
sentinel 0.0 and bootstrap to wrong values (caught by failing G1
test before commit). Seed loop runs at TOP of
`with_controllers_bootstrapped`.
## Diag bake-in
JSONL gains `isv_config` block exposing all 10 design constants per
step:
isv_config.{reward_clamp_win, reward_clamp_loss, kl_target,
improvement_threshold, plateau_patience, div_target,
entropy_target_frac, kurt_lift_scale, ppo_clamp_margin,
lr_warmup_steps}
Post-hoc analysis can correlate any controller's behaviour with the
exact design constants it saw, without grepping the source for
`#define` defaults.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) — skip 10 new design-
constant slots in sentinel-zero loop, assert seeded values
separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
644fbe0348 |
fix(rl): ISV-driven K-loop divisor + max ceiling (slot 450, 451)
f2ggr confirmed K-loop wiring works mechanically but K=8 firing on
22 % of steps over-trained at b_size=1: KL excursions to 12.44
(vs prior 3.4e-4), policy overshoot, reward/trade -$0.585 → -$0.723.
Per `feedback_isv_for_adaptive_bounds` the K-loop config must live
in ISV, not as hardcoded values in the trainer. Two new slots:
RL_K_LOOP_DIVISOR_INDEX (450) — divides n_rollout_steps to get K
Default 2048 (matches ROLLOUT_BOOTSTRAP
so K=1 at controller bootstrap)
RL_K_LOOP_MAX_INDEX (451) — clamp ceiling on K
Default 4 (was hardcoded 8; halved
to prevent gradient overtraining)
K computation in step_with_lobsim now reads both from ISV:
K = clamp(isv[404] / isv[450], 1, isv[451])
Halves worst-case overtraining while preserving the controller
cascade activation (KL above noise floor, ε actively adapting,
ratio_clamp firing). Distribution shifts from K=8 @ 22% → K=4 @ 22%
(half the gradient updates in the high-noise case).
## Wiring
`rl_streaming_clamp_init.cu` extended to seed 5 ISV-resident design
constants (was 3): adv_var_clamp, td_kurt_clamp, adv_var_target,
k_loop_divisor, k_loop_max. Still one kernel call, no HtoD.
## Diag bake-in
JSONL `k_updates` field replaced with `k_loop` block:
k_loop.k_updates — actual K used this step
k_loop.divisor — current divisor (reads isv[450])
k_loop.max — current max (reads isv[451])
Post-hoc analysis can verify the K-computation by independently
recomputing K from isv[404] / k_loop.divisor.
## Slot allocation
RL_SLOTS_END: 450 → 452 (+2 new config slots).
## Test updates
G1 + G3 skip slots 450, 451 in sentinel-zero loop and assert seeded
values (2048.0 + 4.0) separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
1d8ef94848 |
fix(rl): wire n_rollout_steps as K-loop + raise LR_MIN to 1e-4
Two coordinated fixes for the alpha-rl-frt7s findings:
## Issue 1: n_rollout_steps controller was write-only
ISV consumer audit confirmed: 7 of 8 RL controllers had a non-
controller consumer in the per-step path; n_rollout_steps had ZERO.
The controller adapted its output between 256-8192 but nothing read
it. Bit-identical losses between cvf86 and frt7s confirmed: even
fixing the target (0.1 → 5.0) and putting the controller into
healthy HOLD/SHRINK/WIDEN distribution had zero behavioral impact
because no downstream code gated on the emitted value.
### Fix: wire as DQN-replay + PPO+V K-loop multiplier
step_with_lobsim now wraps (sample_and_gather + step_synthetic +
PER priority update) in a K-loop where:
K = clamp(isv[RL_N_ROLLOUT_STEPS_INDEX] / 1024, 1, 8)
Mapping:
* isv[404] = 256 (MIN) → K = 1 (current behavior)
* isv[404] = 2048 (BOOTSTRAP) → K = 2
* isv[404] = 8192 (MAX) → K = 8
Each iteration re-samples PER (different transitions per Adam step)
and runs full Q + π + V forward + backward + Adam. Adapts the
training:env ratio so noisy-advantages regimes get more gradient
samples per env step without slowing env stepping. Directly
addresses the b_size=1 gradient starvation that left l_q stuck at
2.82 in frt7s.
Semantic fit: n_rollout_steps's design intent ("noisy advantages →
need more samples per update") now drives "more training updates
per env step" — equivalent semantics, fits the b_size=1
architecture without requiring a PPO rollout buffer refactor.
`last_k_updates` field tracks the per-step K value for diag.
## Issue 2: LR plateau-decay Q-lock
frt7s deep dive showed:
* Q best=2.3230 locked at step ~783 from a brief downward
excursion during early-training noise
* loss_ema range across 50k steps: [2.323, 3.113]; mean 2.819,
std 0.104
* ZERO steps had loss_ema < best in entire run (let alone <
best × 0.99 = 2.30 threshold)
* 7 LR halvings drove all heads to LR_MIN = 1e-5 by step 7783
* At 1e-5, Q's per-step Adam update is too small to escape;
l_q stayed at ~2.82 for 42k more steps
The plateau-decay is CORRECTLY identifying "model has stopped
improving" — the fix isn't to make plateau detection less
sensitive (loosening threshold to 0.95/0.90 still finds zero
improvements). The fix is to raise the floor LR so the model
has enough learning rate to escape the noise-locked best.
### Fix: LR_MIN 1e-5 → 1e-4 + WARMUP_STEPS 500 → 2000
* LR_MIN raised 10× — even at the plateau-decay floor the model
gets meaningful gradient. Still 10× below LR_BOOTSTRAP=1e-3
so the controller has full dynamic range.
* WARMUP_STEPS raised 4× — gives loss_ema 2000 observations
(≈145 EMA half-lives at α=0.05) to settle BEFORE best is
locked. Prevents the "lucky early excursion locks unreachable
bar" failure mode.
## Diag bake-in
JSONL gains `k_updates` field (per-step K value from the n_rollout
loop) so post-hoc analysis can correlate the K-multiplier with
loss trajectories.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
## Quality-first scope decision
User requested "quality over speed". Considered alternatives:
* Building a proper PPO rollout buffer (Issue 1) — significant
refactor, ~1-2 days. K-loop interpretation chosen instead
because it (a) matches the controller's design intent, (b)
requires no buffer/gradient-accumulation infrastructure, (c)
directly addresses Q learning starvation by giving more
gradient samples per env step.
* Encoder LR decoupling (Issue 2) — encoder receives gradient
from all head backward kernels with their own LRs; treating
the encoder separately would require restructuring all
backward kernels. LR_MIN raise + WARMUP extension gives the
same benefit at the head level without that scope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
95dcc4e312 |
fix(rl): ISV-driven ADV_VAR_RATIO_TARGET for rl_rollout_steps_controller
cvf86 controller_branch diag (commit
|
||
|
jgrusewski
|
708c121f20 |
fix(rl): bounded multiplicative step + noise-floor on rollout_steps + per_α
kc2h9 confirmed: clamping streaming-kernel outputs to [≤100, ≤30]
had ZERO behavioral impact because rl_rollout_steps_controller's
prior design used `scale = clamp(input/target, 0.5, 2.0)` — the
scale saturated to ±2× on the SIGN of (input − target), not the
magnitude. With target=0.1 and typical input=1–10 the controller
slammed to MAX in ≤4 steps regardless of whether input was 4 or
3e5. Bit-identical losses between gxhr8 and kc2h9 confirmed the
saturation.
## Fix 1: rl_rollout_steps_controller — same Schulman pattern as ppo_clip
* input > TARGET × 1.5 → scale = 1.5 (widen)
* input < TARGET / 1.5 → scale = 1/1.5 (shrink)
* in-band → scale = 1.0 (hold)
* input < TARGET × 0.01 → return (noise floor — hold prev)
Per-step adjustment bounded at 1.5×, so rollout_steps drifts
smoothly toward MIN/MAX rather than slamming there. The noise-floor
gate matches the pattern from
`pearl_multiplicative_controllers_need_bounded_step_and_noise_floor`
applied to the ε and τ controllers earlier in R9.
## Fix 2: rl_per_alpha_controller — noise-floor gate (defensive)
per_α uses a LINEAR lift `0.4 + 0.2·(kurt-3)/7` (not multiplicative),
so it doesn't have the saturation bug. But added a noise-floor gate
at KURT_NOISE_FLOOR = 1.0 so a sub-Gaussian kurtosis reading from
the streaming estimator's startup window (when per-step batch-mean
deviations are small before tails develop) doesn't drag α toward
PER_ALPHA_MIN on cold-start.
## Diag bake-in (per user request "bake in diags")
JSONL gains a `controller_branch` block exposing the
multiplicative-controller inputs alongside their design targets:
controller_branch: {
rollout_steps_input: isv[421], rollout_steps_target: 0.1,
ppo_clip_input: isv[419], ppo_clip_target: 0.01,
target_tau_input: isv[418], target_tau_target: 0.01,
per_alpha_input: isv[422], per_alpha_target: 0.6,
}
Post-hoc analysis can compute the branch each step (WIDEN / HOLD /
SHRINK / NOISE) by comparing input/target against the ±33%
tolerance band, revealing whether each controller is being driven
by real signal or sitting in the in-band hold zone. Targets are
reflected from the kernel #defines (synchronised by code review at
the controller-cu file level — there's no ISV slot for these
design constants because they're fundamental to the controller's
behaviour, not adaptive).
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
66115007ab |
fix(rl): ISV-driven output clamp on streaming var/kurtosis kernels
gxhr8 confirmed the streaming kernels work — both formerly-dead
controllers (rl_rollout_steps, rl_per_alpha) now adapt instead of
pegging at MIN. But the unclamped streaming outputs reached
advantage_var_ratio = 3e5 (when streaming-mean passed through zero
and `var/|mean|` blew up under the 1e-6 denominator floor) and
td_kurtosis = 50.6, pegging both downstream controllers at MAX
instead. Per_α at MAX over-concentrates PER sampling on outliers,
which hurts distributional Q learning (best l_q window regressed
from 2.41 → 2.69 between pdgxn and gxhr8).
## Fix: ISV-resident output clamp ceilings
Two new ISV slots hold the streaming-kernel output ceilings:
RL_ADV_VAR_RATIO_CLAMP_INDEX = 447 (default 100.0)
RL_TD_KURTOSIS_CLAMP_INDEX = 448 (default 30.0)
* 100.0 for var_ratio = 1000× ADV_VAR_RATIO_TARGET (= 0.1) — wide
enough that healthy signal (typical 1-10) passes through, tight
enough that 3e5 outliers don't peg rollout_steps.
* 30.0 for kurtosis = 3× (KURT_GAUSSIAN + KURT_LIFT_SCALE) — lets
the full per_α response range engage on heavy-tailed signal
(≤ 10), bounds runaway above that.
Per `feedback_isv_for_adaptive_bounds`: the clamps live in ISV
(visible in diag, modifiable at runtime via re-launching the init
kernel or a future adaptive controller) rather than as kernel-side
`#define`s.
## Seeding (no HtoD per feedback_no_htod_htoh_only_mapped_pinned)
New device kernel `rl_streaming_clamp_init.cu` — single thread,
writes both clamp ceilings directly to ISV. Launched once at the
end of `with_controllers_bootstrapped` alongside the 8 existing
controller-bootstrap launches. Zero host→device transfer.
## Diag bake-in (per user request "ensure to bake in diags")
JSONL gains a new `streaming` block exposing:
* `streaming.adv_var.{mean, m2, clamp}`
* `streaming.td_kurt.{mean, m2, m4, clamp}`
Cross-check: when consumer-input slot (RL_ADVANTAGE_VAR_RATIO_EMA_INDEX
or RL_TD_KURTOSIS_EMA_INDEX) reads exactly the same value as
`streaming.*.clamp`, the clamp fired this step.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert that
ISV[417..END] is sentinel-zero at bootstrap. Both new slots are
seeded to non-zero values by rl_streaming_clamp_init during
bootstrap, so both tests skip these slots in the loop and assert
the seeded values separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
53aeef099b |
feat(rl): ISV-driven PPO importance-ratio clamp + log-ratio diagnostic
pt67l confirmed reward-scale + V-target clamp eliminate V regression
spikes — but exposed a residual: |l_pi| max=586 with mean 0.22. Root
cause: PPO's clip(r, 1-ε, 1+ε) bounds the loss only when surr2 is
the active min. The unclipped branch IS active when A<0,r>1+ε
(surr1=A·r is then more negative than surr2=A·(1+ε), so min selects
surr1) and when A>0,r<1-ε. In the first case `r` can blow up: we've
seen r reach 1e10 from policy drift over a multi-step rollout
producing l_pi=O(1e10) spikes that contaminate the loss-balance
controller and the LR controller's plateau detection.
## Fix: ISV-driven ratio clamp
Per `feedback_isv_for_adaptive_bounds` and
`pearl_controller_anchors_isv_driven`: the clamp ceiling lives in
ISV[RL_PPO_RATIO_CLAMP_MAX_INDEX = 440], not as a hardcoded #define.
New controller `rl_ppo_ratio_clamp_controller.cu`:
* Anchors on the (already KL-adaptive) PPO clip ε at ISV[402]
* target = (1 + ε) × PPO_CLAMP_MARGIN (MARGIN = 10.0)
* Wiener-α blend with floor 0.4 per
pearl_wiener_alpha_floor_for_nonstationary (ε is non-stationary)
* Permanent floor 2.0 / ceiling 1000 per
pearl_blend_formulas_must_have_permanent_floor
* Bootstrap 10.0, replace-directly on first non-bootstrap ε
observation per pearl_first_observation_bootstrap
When ε is small (rl_ppo_clip_controller seeing low KL → tight clip
band), the ratio clamp tightens — outliers should be rare anomalies.
When ε widens (large KL → wide clip band), the clamp widens
proportionally — outliers are expected so we permit more
magnitude before bounding.
## Wiring
ppo_clipped_surrogate_fwd and _bwd both read
isv[RL_PPO_RATIO_CLAMP_MAX_INDEX] and clamp ratio to
[1/ratio_max, ratio_max] before forming surr1/surr2. The clamp is
forward-only in effect (bwd gates pg_grad inside [1-ε, 1+ε] anyway
so gradients were already bounded), but bounding the FORWARD ratio
keeps l_pi sane for the controllers downstream.
The new controller is wired into both:
* `with_controllers_bootstrapped` — bootstrap launch alongside
the other 7 R1 controllers
* `launch_rl_controllers_per_step` — per-step refresh alongside
the other 7 R5 controllers
## Diagnostic: per-step max |log_ratio|
New kernel `ppo_log_ratio_abs_max_b.cu` (same tree-reduce shape as
rl_kl_approx_b) writes per-batch max(|log π_new − log π_old|) to
ISV[RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441]. Launched right after
rl_kl_approx_b (uses the same log_pi_old_d + pi_log_prob_d inputs).
Surfaces in diag JSONL as:
"ppo": {
"ratio_clamp_max": isv[440], # adaptive ceiling
"log_ratio_abs_max": isv[441] # per-step observed max
}
The clamp fires when log_ratio_abs_max > ln(ratio_clamp_max).
For ratio_clamp_max = 10, ln = 2.30. Healthy training has
log_ratio_abs_max well below this most steps; outliers touch or
exceed it on rare excursions which the clamp bounds before they
pollute l_pi.
## Slot allocation
RL_PPO_RATIO_CLAMP_MAX_INDEX = 440 (controller output)
RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441 (per-step diag)
RL_SLOTS_END = 442 (was 440)
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert ISV[417..END]
== 0.0 to catch slot-wiring bugs. Slot 440 is now a controller
OUTPUT bootstrapped to 10.0, so both tests skip it in the loop and
assert == 10.0 separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with new slot-440 assertion)
G3 controllers ✅
G4 target_update ✅
G6 r7d_per_wiring ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
20c7852b66 |
fix(rl): asymmetric clamp on scaled reward + pre-clamp |max| diag
The xv66n smoke (commit
|
||
|
jgrusewski
|
d5c29fb4fa |
fix(rl): warmup window in plateau-decay LR controller fixes V cold-start
`alpha-rl-rzltn` exposed a bug in the plateau-decay design: V head's
`best` got bootstrapped to 7.12e-10 (machine epsilon) at step 1
because V regression had no reward signal yet — no trade had closed,
the bootstrap V target was 0, so the first V loss was effectively 0.
Every subsequent V loss EMA was orders of magnitude higher (4.07
at step 100, 1.15 at step 1000), so the improvement check
`loss_ema < best * 0.99` evaluated false FOREVER. The controller
then decayed lr_v every 1000 steps purely on the patience clock,
not because the model genuinely plateaued.
Cross-check across the 50k-step rzltn run:
* V best unique values: {0.0, 7.12e-10} — ONLY 2 across 50000 rows
* V best max: 7.12e-10
* V best-improvements: 0 (Q: 12, π: 12)
* V decays still fired: 7 (one every 1000 steps from step 1001)
The plateau-decay mechanics worked correctly — the controller counted
to 999 then halved LR exactly as designed. The bug was that "first
observation defines best forever" is degenerate for sparse-signal
heads whose first loss is a cold-start artifact.
## Fix: LR_WARMUP_STEPS
Three new ISV slots (one per head — Q, π, V at 436/437/438) hold a
monotonic warmup counter clamped at LR_WARMUP_STEPS = 500. During
warmup the controller:
* always overwrites `best` with current loss_ema (tracks the EMA
as it converges)
* holds the plateau counter at 0 (no decay fires during warmup)
* increments warmup_counter
Once warmup_counter >= LR_WARMUP_STEPS, the controller switches to
standard plateau detection — `best` then locks in at the
post-warmup loss_ema value (representative of the head's converged
loss scale), and patience counting begins.
At α=0.05 the EMA half-life is ~14 steps; 500 updates leaves ~35
half-lives, well past convergence. This gives V time to see its
first actual losses after trades start closing.
## Slot allocation
RL_SLOTS_END: 436 → 439 (adds 3 warmup counter slots).
## Wiring
* rl_lr_controller.cu — adds warmup_slot param to
plateau_decay_head, kernel takes 12
slot ints (was 9)
* isv_slots.rs — 3 new constants, RL_SLOTS_END += 3
* integrated.rs — launch_rl_lr_controller passes 12
slot ints
* alpha_rl_train.rs — diag JSONL emits new
lr_plateau.{head}.warmup field
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
G6 r7d_per_wiring ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
13d81dc5e6 |
diag(rl): emit grad_norm_ema + lr_plateau state in alpha_rl_train JSONL
Adds two new top-level keys to each diag.jsonl row:
"grad_norm_ema": {q, pi, v} — slots 424-426
"lr_plateau": {q,pi,v} × {loss_ema, best, stale} — slots 427-435
With these in place we can independently verify each plateau-decay
event in `mjgsj`'s diag (and all future runs):
* `loss_ema` traces the controller's slow EMA of head loss
(α=0.05); confirms the EMA actually moves and isn't stuck on the
bootstrap zero
* `best` shows the rolling minimum the controller compares against;
confirms it improves early then plateaus
* `stale` is the steps-since-best counter; should hit
PLATEAU_PATIENCE = 1000 exactly when an LR halving fires; reset to
0 after every decay event or every improvement
The `grad_norm_ema` block is kept because the grad-norm producers are
still wired (commit
|
||
|
jgrusewski
|
87a22d12c9 |
feat(rl): walk-forward G8 eval phase + fold split (MVP, manual fan-out)
Adds the minimum-viable implementation of the R9 multi-fold G8 gate
per `pearl_single_window_oos_is_not_oos` ("a single window is NOT
out-of-sample"). The trainer can now:
1. Slice the MBP-10 file list into K equal-sized blocks
(`--n-folds K --fold-idx k`).
2. Train on blocks [0..=k] (passed to MultiHorizonLoader).
3. Run a separate eval phase of `--n-eval-steps` on block [k+1]
using a second loader instance.
4. Drain LobSim trade records gated by a pre-eval head checkpoint
so train-phase trades don't contaminate the eval summary.
5. Compute profit_factor + sharpe + drawdown via existing
`ml_backtesting::artifacts::compute_summary`.
6. Write `eval_summary.json` alongside `alpha_rl_train_summary.json`.
## Manual fan-out (this MVP)
The dispatcher (`scripts/argo-alpha-rl.sh`) gains three new flags
that thread through the Argo template into the CLI: `--fold-idx`,
`--n-folds`, `--n-eval-steps`. To run a 3-fold G8:
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 0 --n-eval-steps 200
./scripts/argo-alpha-rl.sh --n-folds 3 --fold-idx 1 --n-eval-steps 200
(With n_folds=3 the valid fold indices are 0 and 1 — the third block
is the eval window for fold 1. n_folds=K accepts fold_idx ∈ [0, K-2].)
Each submission produces one `eval_summary.json` at the resolved
output dir; the per-fold profit_factor is the value to aggregate.
Manual aggregation for now — automated DAG matrix fan-out + an
in-cluster aggregator pod is a follow-up commit. The aggregator
will mean ± SD the per-fold PFs and gate on `PF > 1.0`.
## What's NOT pure eval
The eval loop calls `step_with_lobsim` (same as train) — Adam steps,
PER updates, controller adaptations all still fire during eval. At
b_size=1 the per-step learning effect is small relative to the
train-phase-accumulated policy, so the eval PF approximates the
OOS performance of the train-end policy. A clean pure-eval mode
(forward + LobSim step only, no backward/Adam/PER) is a follow-up
architectural change; documented inline at the eval phase block.
## Default behaviour unchanged
`--n-folds=1` (default) skips the eval split entirely and uses all
files for training — identical to the prior single-window smoke.
The R9 prior smokes ran in this mode. Default `--fold-idx=0` and
`--n-eval-steps=0` keep prior smoke runs binary-compatible.
## Template + dispatcher changes
* `alpha-rl-template.yaml`: adds 3 new workflow parameters
(`fold-idx`, `n-folds`, `n-eval-steps`) and threads them into
the train container's `alpha_rl_train` invocation.
* `argo-alpha-rl.sh`: adds matching CLI flags with explicit
documentation of the multi-fold dispatching pattern.
## Verified gates
Local sm_86 build + dispatcher syntax clean. Tests unchanged
(the walk-forward path is exercised by cluster smokes, not unit
tests — the loader-slicing logic is straightforward index math).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
ce1e13519b |
fix(rl): mapped-pinned for all R7d/R8 CPU↔GPU paths + diag JSONL + guard
Two concerns in one commit since they're entangled:
## 1. feedback_no_htod_htoh_only_mapped_pinned violations
R7d (PER push/sample) + R8 (CLI binary) + the new per-step diag dump
shipped with 8 raw `stream.memcpy_htod` / `stream.memcpy_dtoh` calls.
The rule is explicit: "mapped-pinned only for CPU↔GPU; tests not
exempt." A raw `stream.memcpy_*` on a regular `&[T]` / `&mut [T]` is
NOT mapped-pinned — the source/dest slice isn't page-locked, so the
CUDA driver does an internal blocking HtoD/DtoH that stalls the
stream.
Refactored all 8 violations to use the mapped-pinned + DtoD pattern
(cuMemHostAlloc DEVICEMAP — host writes via `host_ptr`, kernel reads
`dev_ptr`, DtoD between them via `cudarc::driver::result::memcpy_dtod_async`).
New shared helpers in `trainer/integrated.rs`:
* `read_slice_i32_d` — DtoH for `i32` device buffers via
`MappedI32Buffer` staging. Counterpart to the existing
`read_slice_d` (f32 version).
* `write_slice_f32_d` — CPU→GPU upload for `f32` via
`MappedF32Buffer.write_from_slice` + DtoD into destination.
* `write_slice_i32_d` — CPU→GPU upload for `i32` via
`MappedI32Buffer.host_slice_mut().copy_from_slice` + DtoD.
`pub fn` wrappers (`read_slice_*_d_pub`) expose the f32/i32 helpers
to the CLI binary so the per-step diag DtoH uses the same canonical
pattern.
Call-site refactors:
* `push_to_replay`: 4× `stream.memcpy_dtoh` → `read_slice_*_d`.
* `sample_and_gather`: 3× `stream.memcpy_htod` → `write_slice_*_d`.
* `step_with_lobsim` pre-PER ISV refresh: raw `memcpy_dtoh` →
`read_slice_d` (424 floats per step).
* `step_with_lobsim` post-Q PER priority TD readback: raw
`memcpy_dtoh` → `read_slice_d` (b_size floats per step).
* `step_synthetic` ISV mirror refresh: raw `memcpy_dtoh` →
`read_slice_d` (pre-existing pre-R9 violation; fixed in the
same commit since it's the same pattern in the same file).
* Init-time (one-shot) `prng_state` upload: raw `memcpy_htod` →
inline mapped-pinned DtoD (custom because cast through i32 for
the u32 buffer).
* Init-time (one-shot) `atom_supports` upload: raw `memcpy_htod`
→ `write_slice_f32_d`.
* `examples/alpha_rl_train.rs` per-step diag DtoH (3 calls) →
`read_slice_*_d_pub`.
## 2. Pre-commit guard gap — diff-aware HtoD/DtoH check
The existing GPU hot-path guard (`scripts/gpu-hotpath-guard.sh`)
EXPLICITLY skips memcpy_htod/dtoh on the assumption that such calls
only appear in `cuda_pipeline/` (where mapped-pinned is the
convention). That assumption was falsified by R7d/R8 — the guard
shipped 8 violations green.
Added `check_no_raw_htod_dtoh` to `scripts/pre-commit-hook.sh` (the
real file behind the `.git/hooks/pre-commit` symlink). The check is
DIFF-AWARE: it greps only the `+` lines of `git diff --cached -U0`,
so pre-existing violations elsewhere (143 sites across the codebase)
don't block commits touching unrelated files. NEW additions of
`\.memcpy_(htod|dtoh)\(` are flagged with a clear error pointing at
the mapped-pinned alternative. Suppress per-line with `// gpu-ok:
<reason>` (same convention as the existing guards).
Pre-existing violations in `ml-alpha/src/aux_heads.rs`,
`mamba2_block.rs`, `cfc/`, `data/`, etc. are a separate cleanup —
not blocked by this commit's check because the diff-aware filter
ignores anything that was already on `HEAD~1`.
## Verified gates (post-fix, local sm_86)
G1 isv_bootstrap ✅ unchanged
G3 controllers_emit ✅ unchanged
G4 target_soft_update ✅ unchanged
G6 r7d_per_wiring ✅ unchanged (PER round-trips all
mapped-pinned now)
R3 ema/advantage (3 tests) ✅ unchanged
R4 action kernels (3 tests) ✅ unchanged
end integrated_trainer_smoke ✅ unchanged
Mapped-pinned is semantically equivalent to raw memcpy_htod/dtoh —
just routed through page-locked staging so the driver doesn't have
to do its own internal pinning. Behaviour identical; the cost shifts
from "driver hidden HtoD per call" to "mapped-pinned alloc + DtoD
per call." For the smoke (b_size=1, 1000 steps), the cost difference
is in the microseconds.
## Per-step diag JSONL (separate concern, same commit)
Added `--diag-jsonl <PATH>` flag to `alpha_rl_train.rs` (default:
`<out>/diag.jsonl`). After each `step_with_lobsim`, writes one JSON
record capturing:
* step number, elapsed wall time
* all 5 head losses + λs
* all 7 RL controller outputs (γ τ ε coef n_roll per_α scale)
* all 5 per-head learning rates (lr_bce/q/pi/v/aux)
* all 7 EMA inputs the controllers consume
* replay buffer length
* per-step reward stats (sum, max, min, abs_max)
* per-step done count
* per-step action histogram (9 action classes)
Critical for cluster smoke debugging — the prior CLI only flushed
an `eprintln` progress line every N steps (default 100), making
in-flight controller drift / replay stagnation / reward explosion
invisible until they produced a NaN abort. The JSONL is line-
buffered + flushed every `log_every` steps so `tail -f` shows
progress live.
The stderr progress line is also beefed up to include γ / ε / per_α /
reward_scale / dones / rew_sum at each tick so a casual `argo logs`
inspection sees the controller behaviour without parsing JSONL.
## Why R9 cluster submission needs this
Without the diag dump, an R9 1000-step smoke is "blind" — only the
final summary tells us what happened. With the dump, post-hoc
analysis can answer:
* Did the controllers adapt or stay at bootstrap?
* Did the reward scale stabilise or saturate?
* Did the PER buffer fill?
* Was the action histogram dominated by any one action?
* Where did the per-head losses converge to?
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
jgrusewski
|
1168f3ea83 |
feat(rl): R8 — alpha_rl_train CLI + Argo template + dispatcher
Closes the rebuild plan's R8 scope: production runner shape for the
integrated RL trainer. Three artifacts wired end-to-end:
1. `crates/ml-alpha/examples/alpha_rl_train.rs` — clap CLI driving
`IntegratedTrainer::step_with_lobsim` against MBP-10 windows
loaded via `MultiHorizonLoader::next_sequence_pair` (R2) for
true `(s_t, s_{t+1})` adjacency. Per `feedback_mbp10_mandatory`,
`--mbp10-data-dir` is required — no synthetic-data fallback in
the production path.
2. `infra/k8s/argo/alpha-rl-template.yaml` — WorkflowTemplate
mirroring alpha-perception's DAG (check-cache → ensure-binary →
train; warmup-gpu parallel). Binary cache slot is
`/data/bin/<sha>/alpha_rl_train` (distinct from `alpha_train`
so the two binaries coexist at the same SHA).
3. `scripts/argo-alpha-rl.sh` — dispatcher with three rebuild-plan
guards baked in.
## Dispatcher guards (per the rebuild plan's feedback list)
`feedback_default_to_l40s_pool` (2026-05-09): default `--gpu-pool`
is `ci-training-l40s` (sm_89). H100 (sm_90) is opt-in for production
scale-up only. Cubins must match the device, so the dispatcher
derives `cuda-compute-cap` from the pool name and threads it into
the workflow params.
`feedback_argo_template_must_apply` (2026-05-21 canonical incident):
`argo submit --from=wftmpl/<name>` reads the cluster CRD, NOT the
on-disk YAML; unknown `-p` parameters silently no-op without a prior
`kubectl apply`. Dispatcher applies the local template BEFORE every
submission (overrideable via `--skip-template-apply` for the rare
case where you've already applied manually).
`feedback_push_before_deploy` (2026-05-20 canonical incident): the
in-cluster `ensure-binary` pod fetches source from `origin/<branch>`,
NOT the local working tree. Submitting before `git push` deploys the
last-pushed SHA, which can lag local diff by N commits. Dispatcher
verifies `git rev-parse HEAD == git rev-parse origin/<branch>` and
hard-errors with the explicit push command otherwise. Bypass via
`--skip-push-check` (only when intentionally deploying a previously-
pushed SHA via `--sha`).
## CLI: gate G8 (NaN abort)
Per `feedback_stop_on_anomaly` + `feedback_kill_runs_on_anomaly_quickly`,
the CLI checks every per-head loss (l_bce / l_q / l_pi / l_v / l_aux
/ l_total) for finiteness after each `step_with_lobsim` call.
Non-finite at any step → write summary with `nan_abort_step` set →
`process::exit(2)`. R9's cluster smoke tail-watcher kills the
workflow on the non-zero exit code, satisfying gate G8 from the
rebuild plan.
## CLI: knobs that ARE on the CLI
Structural / boundary parameters only (per
`pearl_controller_anchors_isv_driven`: every adaptive knob lives in
ISV, not CLI flags):
* `--mbp10-data-dir / --predecoded-dir / --out` — I/O paths.
* `--n-steps` — wall-budget control (1000 R9 smoke / 50k+ prod).
* `--seq-len / --n-backtests / --per-capacity` — structural
sizing. seq_len threads into the loader's multi-resolution
`1:<seq_len>` config; n_backtests into both LobSimCuda and
PerceptionTrainerConfig.n_batch.
* `--seed` — reproducibility per
`pearl_scoped_init_seed_for_reproducibility` (forks deterministic
sub-seeds for dqn / ppo / per).
* `--instrument-mode` — MBP-10 filter (all / front-month / id=N).
* `--gpu-idx` — CUDA device selection.
What's NOT on the CLI: γ / τ / ε / entropy_coef / per_α / reward_scale
/ per-head LRs — all live in ISV[400..417] and are driven by R5's
controllers from EMA-tracked diagnostics. Per the rebuild plan
A1: "every adaptive bound is signal-driven, not tuned."
## Cluster smoke entry point
```bash
# R9 validation smoke (after pre-cluster local CUDA tests green).
./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode front-month
# Production scale-up (gated by R9's multi-fold pass).
./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 \
--per-capacity 100000 # GPU sum-tree R-future when capacity > 4096
```
## What's NOT in this commit
The R9 cluster smoke run itself is out of band — this commit ships
the entry points. R9 will execute the pre-cluster validation
checklist + first 1000-step smoke + multi-fold walk-forward G8 gate
per the rebuild plan §"Cluster smoke discipline".
The summary JSON's schema is intentionally narrow (final-step losses
+ replay len + completion state + NaN abort marker). R-future may
add per-epoch breakdowns + per-ISV-slot snapshots once the cluster
smoke tells us which diagnostics are actually load-bearing for kill
decisions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|