Loss minimization (wr=0.561 but PnL negative — losses 32% > wins):
A1: Exempt FlatL/FlatS from confidence gate — exit actions never
blocked, model can always close losing positions.
A2+A3: New rl_drawdown_stop kernel — per-step drawdown penalty
(min(0, unrealized_r) × rate) creates continuous exit gradient.
Hard stop-loss force-closes when unrealized_r < -threshold.
Both ISV-driven (slots 586, 587).
A4: Adaptive LOSS clamp — tracks observed neg/pos EMA ratio instead
of static 3.0. LOSS = clamp(1.0, ratio×1.1, 3.0). Q sees
accurate loss magnitudes.
Performance:
B0: Remove gratuitous stream.synchronize() in apply_snapshot
(sim/mod.rs) — same-stream ordering makes it unnecessary.
Expected: -12-49ms/step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause: confidence gate decouples policy from actions. Policy
softmax stays high-entropy (~2.4) while the gate forces Hold, producing
action entropy ~0.7. SAC read policy entropy EMA (slot 420), saw
"above target", and kept LOWERING τ — the exact opposite of what was
needed.
New kernel `action_entropy_per_step` computes H(action_histogram) from
the POST-gate actions buffer and writes to ISV slot 583
(RL_ACTION_ENTROPY_EMA_INDEX). SAC co-tuning in rl_q_pi_distill_grad
now reads this slot. Launched OUTSIDE CUDA Graph capture (after
confidence gate + FRD gate) in both training and prefill paths.
Kernel design: 11 threads (N_ACTIONS), each thread counts its action
across all b_size elements. Thread 0 computes entropy from the
histogram and updates the EMA. No atomics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add rl_kl_reference_grad kernel: β × (π_θ(a) - π_ref(a)) gradient
fires on ALL batch elements, ALL steps. π_ref = [Hold=50%, rest=5%]
encodes surfer philosophy as a continuous prior.
Unlike entropy (pushes to uniform, doesn't know Hold is special),
gates (binary, blocks learning), or hold-prior advantages (overwhelmed
by PPO ratio), KL penalty is smooth, continuous, and specifically
preserves the Hold-heavy reference distribution.
β=0.5 (ISV slot 578). Hold=75-100% through 5000 steps.
wr=0.339 — profitable trades while maintaining Hold dominance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upload all pre-converted snapshots + FRD labels to GPU at init (1.2GB
for 5.6M snapshots / 2 files locally, ~11GB for 45M / 9 files on L40S).
New GPU kernels: gpu_sample_and_gather (PRNG sampling + AoS-to-SoA),
gpu_gather_next/gpu_gather_current (anchor offset re-gather),
gpu_gather_frd_labels (per-horizon label gather).
step_with_lobsim_gpu: GPU-only data path replacing host-side loader.
Encoder forwards via forward_encoder_from_device. Eliminates 7700 heap
allocs + 418k scalar copies + 16ms CPU work per step at b=256.
Init uploads use cuMemcpyHtoD_v2 (synchronous, one-time).
Note: apply_snapshot skipped (lobsim book stale, dones/rewards=0).
Follow-up: copy last-snapshot book data from SoA to lobsim buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New snapshot_aos_to_soa.cu kernel: one thread per snapshot, reads
contiguous Mbp10RawInput structs from mapped-pinned AoS buffer,
scatters fields to 10 SoA device buffers. Replaces 418k scalar
host copies + 10 DtoD memcpys with single memcpy + one kernel launch.
Add #[repr(C)] to Mbp10RawInput with 216-byte compile-time assertion.
Single MappedRecordBuffer replaces 10 separate staging buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace fatbin with single-arch cubin auto-detected via nvidia-smi.
Each node compiles for its own GPU — faster builds, guaranteed compat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace -cubin -arch=sm_XX with -fatbin -gencode for all 3 GPU archs.
All include_bytes! paths updated from .cubin to .fatbin. Removes
CUDA_COMPUTE_CAP env var dependency. cuModuleLoadData transparently
selects the right arch from the fat binary at runtime.
Fixes CUDA_ERROR_NO_BINARY_FOR_GPU on H100 (sm_90).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wire outcome CE backward → grad_W/b → Adam → grad_h_accumulate.
Add PopArt, spectral, Q-bias, per-branch LR, outcome metrics to
diag JSONL for training validation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backward HER: on trade close, injects synthetic with peak PnL if
peak > 1.5× actual. Forward HER: evaluates closed trades after 50
steps, injects if holding would have been better.
ISV bootstrap: threshold=1.5, priority_boost=3.0, lookahead=50.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 kernels for all-device prioritized experience replay:
- rl_per_push: n-step accumulation + single-block prefix-sum for
write_head coordination (no atomicAdd)
- rl_per_sample: stratified proportional sampling via top-down
sum-tree walk with xorshift32 PRNG + inline gather
- rl_per_update_priority: write |TD|^α to leaves + shared-mem
block-wide max reduction
- rl_per_tree_rebuild: bottom-up parallel scan with __threadfence
between levels (15 passes for capacity=32768, no atomics)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Inline launch_l2_norm, launch_ema_update_per_step, launch_kurtosis,
launch_var_over_abs_mean at all call sites to eliminate .clone() that
allocated device memory inside CUDA graph capture regions (caused
CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED at b>16).
- Delete the now-unused helper methods (no hiding, no suppressing).
- Add pre-commit hook guard for *_d.clone() patterns.
- Add rl_fused_reward_pipeline.cu (7→1 per-batch kernel, not yet wired).
- Add rl_write_u64 kernel + ts_ns device-resident for Graph B.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move all step_synthetic/dqn_replay_step alloc_zeros to persistent
trainer fields (ss_* prefix). Enables CUDA Graph capture of the replay
training step — all device pointers are now stable across steps.
Introduces reduce_axis0_free() to resolve borrow-checker E0502 when
both source (per-batch scratch) and destination (reduced grad) are
self fields passed to the same function.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move tau sampling and factored noise generation from host-side ChaCha8
RNG + mapped-pinned upload to device-side xorshift32 kernels. This
eliminates all host-side RNG from the step pipeline, unblocking CUDA
Graph capture for Graphs A and C.
New kernels:
- rl_sample_tau: per-batch xorshift32 generates tau [B, N_TAU] ~ U(0,1)
- rl_sample_noise: factored noise f(x)=sign(x)√|x| for NoisyLinear
Both kernels self-seed from alloc_zeros on first call (zero memcpy).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the IQN integration:
- rl_iqn_backward.cu: backprop through quantile embedding + output
projection to produce per-batch weight gradients for all 4 weight
tensors. Block per batch, HIDDEN_DIM threads.
- rl_iqn_loss.cu (already existed): quantile Huber loss feeds
grad_online_q into the backward kernel.
- rl_ensemble_action_value.cu: E_ensemble = α×C51 + (1-α)×IQN,
ISV-driven α at slot 544.
- Adam updates for IQN weights after backward.
- Ensemble Q feeds rl_q_pi_agree_b diagnostic.
Full stack smoke: 200 steps, l_q=2.43, no crash. Both C51 and IQN
learning simultaneously from the same replay transitions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the Q-learning improvements spec. Adds an Implicit
Quantile Network head alongside the existing C51 head:
- rl_iqn_forward.cu: quantile embedding φ(τ) = ReLU(W × cos(iπτ)),
element-wise h_t ⊙ φ(τ), action-value projection. Plus
rl_iqn_expected_q for tau-mean reduction.
- rl_iqn_loss.cu: quantile Huber loss ρ_τ(δ) = |τ-1(δ<0)| × Huber(δ).
Block tree-reduce per batch (no atomicAdd).
- rl/iqn.rs: IqnHead struct with online + target weights, Xavier init,
forward/forward_target/expected_q/compute_loss methods.
ISV slots: 543 N_TAU (32), 544 ensemble_alpha (0.5), 545 LR (1e-3).
Not yet wired into the trainer — head is constructible and kernels
compile. Ensemble integration is the next step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Overrides closing actions (a3/a4/a9/a10) to Hold when
steps_since_done < RL_MIN_HOLD_STEPS_INDEX (slot 536, default 20).
The agent CANNOT exit before the minimum — forced to ride the wave.
Trail stops still fire regardless (safety overrides patience) —
if the market moves against the position past the trail distance,
the stop-loss exits even within the min-hold window.
Pipeline order: action selection → confidence gate → FRD gate →
min_hold_check → trail_mutate → trail_stop → heat_cap →
actions_to_market_targets
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three ISV-driven reward shaping components discourage churning and
reward patience:
1. Entry cost ($15 default): subtracted when opening a new position.
Agent must expect profit > cost to justify entry. Slightly above
ES 1-tick spread ($12.50) so marginal trades are net-negative.
2. Short-hold penalty (0.5× for holds < 20 steps): multiplicative
penalty at trade close for quick flips. "Don't bail on the first
bump" — halves the reward for sub-5-second holds.
3. Hold bonus ($0.50/step × sqrt(hold_time)): per-step reward for
staying in a profitable position. "Ride the wave" — incentivizes
patience when the trade is working.
Runs BEFORE reward_scale so all costs/bonuses are in raw USD terms.
ISV slots 532-535. RL_SLOTS_END → 536.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_gate_threshold_controller.cu watches dones EMA and adjusts
confidence + FRD gate thresholds via Schulman bounded step:
- dones_ema < target×0.5 → relax thresholds (×0.95)
- dones_ema > target×2.0 → tighten thresholds (÷0.95)
ISV-driven: target (0.02), conf bounds (0.001-0.50), FRD bounds
(0.05-0.50), adjust rate (0.95). Runs per step after warmup.
Also lowers default thresholds: conf 0.10→0.01, FRD 0.35→0.15.
Post-warmup, the controller adapts these based on actual trade
flow instead of relying on static defaults.
ISV slots 525-531. RL_SLOTS_END → 532.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces ENCODER_INPUT_DIM = 56 (FEATURE_DIM + 16). All encoder
first-layer weight matrices (VSN gate, Mamba2 L1 input projection)
now sized for 56 input dims. The extra 16 are per-batch state:
4 trade_context + 12 multires features.
- snap_feature_assemble_batched: output stride → ENCODER_INPUT_DIM,
zero-fills dims [40..56] for the broadcast kernel to overwrite.
- New rl_encoder_context_broadcast.cu: writes trade_context_d[B×4]
+ multires_output_d[B×12] into each of the K sequence rows per
batch at positions [40..56].
- CfcConfig.n_in, Mamba2 L1 in_dim, VSN gate, window_tensor_d,
all forward/backward scratch buffers updated to ENCODER_INPUT_DIM.
- CfcTrunk default config updated.
The broadcast kernel launch integration into the forward_only path
is the final wire-up step — until then dims 40-55 are zero-filled
(safe: Xavier init on new columns means encoder starts by learning
to ignore them, then gradually incorporates the signal).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P1: New rl_trade_context_update.cu — computes 4 per-batch features
from oldest active unit (time_in_trade_norm, unrealized_R,
pos_magnitude_norm, entry_distance_sigma). Output in
trade_context_d[B×4], updated after unit_state_update each step.
P0: New rl_multires_features_update.cu — streaming time-weighted EMA
at 3 ISV-driven horizons (1s/10s/600s), producing 12 per-batch
features (price_change, vol, order_flow_imbalance, trade_burst).
O(1) state per feature vs circular buffer — same time-constant
semantics.
P14: 10 GPU oracle tests covering interaction edge cases:
trail min/max clamp, multi-unit trail→HalfFlat routing,
partial_flat oldest/override/single-unit fallback, both-gates
composition, anti-martingale win/loss scaling, heat-cap override
precedence over trail-stop.
ISV slots: 521-523 (multires horizons). RL_SLOTS_END → 524.
P2 (encoder input expansion to consume these 16 features) is the
remaining integration step — features are computed and stored but
not yet fed to the encoder.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
P10: New rl_recent_outcome_update.cu — per-batch signed outcome EMA
(sign(reward) on done steps) feeds per-batch anti-martingale
sizing in actions_to_market_targets. Replaces the single ISV
scalar with a per-batch buffer for multi-batch granularity.
P11: Trail bootstrap switched from vwap × 1e-3 × k_init to
k_init × MEAN_ABS_PNL_EMA (slot 423). Vol-derived trail
distance adapts to realized trade magnitude as the EMA updates.
P12: P_MIN in rl_pi_action_kernel now ISV-driven (slot 519,
default 0.015). At N=11, max single-action prob = 0.85
(uplift vs prior 0.80 at hardcoded P_MIN=0.02).
ISV slots: 519 P_MIN, 520 OUTCOME_ALPHA. RL_SLOTS_END -> 521.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_frd_gate.cu kernel reads the FRD head's horizon-2 (medium,
~300 ticks) categorical distribution. For long openings, sums
probability mass in the positive tail (atoms > +0.5σ); for short
openings, sums the negative tail (atoms < -0.5σ). Overrides to Hold
when favorable mass < threshold.
Fires after confidence gate, before trail/heat/market pipeline.
Same preconditions: only gates flat positions with opening actions.
ISV slots: 516 THR_LONG (0.35), 517 THR_SHORT (0.35),
518 fired_count (diag). RL_SLOTS_END → 519.
GPU oracle test: 4 cases (uniform pass, peaked-negative gate for
long, peaked-positive gate for short, non-flat bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New rl_confidence_gate.cu kernel computes C51 distributional Lower
Confidence Bound (μ - λσ) / σ_norm for the chosen action. When
position is flat and the selected action is an opening (a0/a1/a5/a6),
overrides to Hold if conf < threshold.
Fires after π action selection, before trail/heat/market pipeline.
Only gates on flat positions — existing positions pass through
unconditionally regardless of Q uncertainty.
ISV slots: 512 threshold (0.10), 513 λ (1.0), 514 σ_norm (1.0),
515 fired_count (diag). RL_SLOTS_END → 516.
GPU oracle test: 4 cases (low-conf gate, high-conf pass, non-flat
bypass, non-opening bypass).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Third and final FRD backward stage. Closes the chain from
softmax+CE loss back to the encoder's hidden state h_t.
Kernel `cuda/rl_frd_layer1_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (HIDDEN_DIM=128, 1, 1)
* Phase 0: threads 0..63 stage dL/dpre_hidden = grad_hidden ×
1{hidden > 0} into shared mem (the cached post-ReLU `hidden`
buffer encodes the mask — hidden == 0 ⇔ pre-activation was
≤ 0 → ReLU killed it). Same thread also writes db1_per_batch.
* Phase 1: each thread k (k < 128) writes one row of
grad_W1_per_batch[b, k, 0..64] (64 writes per thread, no atomics)
* Phase 2: same thread computes grad_h_t[b, k] =
Σ_i W1[k, i] × dL/dpre_hidden[b, i]
* Per-(b, k, i) sole-writer per feedback_no_atomicadd
Rust wiring `FrdHead::layer1_bwd` — takes h_t, hidden (forward cache),
grad_hidden (from layer2_bwd), self.w1_d; writes grad_w1_per_batch,
grad_b1_per_batch, grad_h_t. The grad_h_t buffer becomes the encoder-
upstream gradient that the trainer's grad_h_accumulate kernel folds
into the encoder's gradient with λ_frd scaling (same pattern as Q/π/V
heads — wiring lives in F.4).
Tests (2 new, 10/10 file total):
* frd_layer1_bwd_finite_diff_w1 — perturbs the W1 slot with MAX
|analytical gradient| (instead of an arbitrary fixed slot — fp32
finite-diff is rounding-error-limited so a tiny gradient gives
misleading rel_err). At max-magnitude slot (k=84, i=55): analytical
= -0.0451, numerical = -0.0448, rel_err = 5.6e-3 — well within
1e-2 tolerance (slightly looser than dW2's 5e-3 because dW1
crosses an extra matmul + the ReLU mask boundary).
* frd_layer1_bwd_relu_mask_zeros_grad — fixture with h_t = all -1
produces ~half the hidden slots ReLU-masked (cached hidden = 0).
For every masked slot i, asserts:
* db1_per_batch[b, i] == 0 (exact equality — mask is hard 0)
* dW1_per_batch[b, k, i] == 0 for every k (~32 × 128 = 4096
slots checked)
Empirically 32/64 masked, 32/64 active — confirms ReLU mask
is wired through the chain correctly without leaking gradient
through dead branches.
F.3 backward chain is now complete end-to-end:
rl_frd_softmax_ce_grad (F.3a) → rl_frd_layer2_bwd (F.3b) →
rl_frd_layer1_bwd (F.3c) → grad_h_t (consumed by F.4 wiring)
F.4 wires Adam optimizers for W1/b1/W2/b2 + grad_h_accumulate into
the encoder gradient + loader-side label generation + λ_frd × CE
into stats.l_total.
Second of three FRD backward stages. Given dL/dlogits from F.3a's
softmax_ce_grad and the cached hidden activation from F.2's forward,
computes the layer-2 weight gradients via the standard chain rule
and emits the upstream gradient for layer-1 backward (F.3c).
Kernel `cuda/rl_frd_layer2_bwd.cu`:
* grid_dim = (B, 1, 1), block_dim = (FRD_HIDDEN_DIM=64, 1, 1)
* Phase 0: stage 63-slot grad_logits into shared (thread 63 idle)
* Phase 1: each thread i (i < 64) computes one row of per-batch
dW2 scratch: grad_w2_per_batch[b, i, 0..63] = h_bi × grad_logits[0..63]
(63 writes per thread, no atomics)
* Phase 2: each thread i computes dL/dhidden[b, i] = Σ_j W2[i, j] × grad_logits[j]
* Phase 3: thread i (i < 63) writes grad_b2_per_batch[b, i] = grad_logits[b, i]
* Per-batch scratch shape [B, FRD_HIDDEN_DIM, FRD_OUT_DIM] reduces
across batch via existing reduce_axis0 infra (caller's job, same
pattern as v_head_bwd / aux_heads_bwd)
Rust wiring `FrdHead::layer2_bwd`:
* Takes hidden (forward cache), grad_logits (from softmax_ce_grad),
self.w2_d
* Writes grad_w2_per_batch, grad_b2_per_batch, grad_hidden — all
sized to caller-allocated buffers
* Sole &self method (Adam step is the caller's responsibility)
Tests (2 new, 8/8 file total):
* frd_layer2_bwd_finite_diff_w2 — perturb W2[10, 5] by ±ε=1e-3,
compare (L(+) - L(-))/(2ε) to per-batch grad scratch. rel_err
= 6.27e-5 (better than F.3a's softmax-CE finite-diff because
the gradient magnitude here is larger so rounding error is
relatively smaller). Helper `ce_total_loss` re-uses
`softmax_ce_grad` to compute total CE for the perturbed forward
pass — pure GPU-oracle, no CPU softmax/CE reference impl.
* frd_layer2_bwd_db2_equals_grad_logits — analytical invariant:
db2_per_batch[b, j] must equal grad_logits[b, j] exactly (the
bias gradient is the identity passthrough at this layer). Cheap
structural check that catches dimension-shuffle bugs in the
kernel before they corrupt the reduce_axis0 step.
The kernel restores W2 to its original values after the perturbation
to keep test isolation clean — `&mut head` access pattern (proper
Rust borrowing, no UB const→mut casts).
F.3c (layer-1 backward: dW1, db1, dh_t with ReLU mask via the
cached hidden activation) is next.
Per-(batch, horizon) softmax + cross-entropy loss + gradient w.r.t.
the 21 atom logits. First of three backward stages — F.3b adds layer-2
weight grads (dW2, db2, dhidden), F.3c adds layer-1 weight grads
(dW1, db1, dh_t with ReLU mask).
Kernel `cuda/rl_frd_softmax_ce_grad.cu`:
* grid_dim = (B, FRD_N_HORIZONS, 1), block_dim = (FRD_N_ATOMS=21, 1, 1)
— one block per (batch, horizon) pair, threads cooperate over the
21 atoms via shared mem
* Standard numerically-stable softmax: shift by row_max, exponentiate,
normalize by row_sum (thread 0 does the serial reductions — 21
atoms is small enough warp-shuffle overhead isn't worth it)
* Gradient: (p[a] - 1{a==label}) / B at the source per v_head_bwd
convention (mean-reduce over batch)
* Loss: -log(p[label]) with 1e-30 floor against log(0)
* Sentinel label (-1) zeros both gradient row and loss — for the
missing-horizon case at the rightmost edge of the snapshot stream
(forward returns at h=300 ticks aren't realized for the last
300 snapshots; loader marks those labels with -1)
* Per feedback_no_atomicadd: per-(b, h, a) sole-writer pattern
Rust wiring `src/rl/frd.rs::FrdHead::softmax_ce_grad`:
* Second cubin loaded alongside fwd (separate module per the
aux_heads pattern; small handle, no impact on init time)
* Caller provides labels_d [B, FRD_N_HORIZONS] of i32 and gets back
grad_logits + per-(b, h) raw CE; sum + λ_frd scaling left to the
caller (F.4 will hook this into stats.l_total + Adam step)
Tests `tests/frd_head.rs` — 3 new GPU-oracle tests (6/6 file total),
all PASS on RTX 3050 Ti:
1. frd_softmax_ce_grad_uniform_logits_match_log_n_atoms — for any
label, uniform logits → CE = ln(FRD_N_ATOMS) = ln(21) ≈ 3.0445.
Also asserts per-row Σ grad_logits = 0 (softmax-CE invariant).
2. frd_softmax_ce_grad_sentinel_label_zeros_row — label=-1 with
non-trivial random logits produces exactly zero loss + grad
for every row (no leak through the sentinel path).
3. frd_softmax_ce_grad_finite_diff_matches_analytical — perturbs
one logit slot by ±ε=1e-3, compares (L(+ε) - L(-ε))/(2ε) to
the kernel's analytical gradient. rel_err ≈ 1.3e-3 (fp32
finite-diff is rounding-error-limited at this ε; tolerance
set to 5e-3 with explanatory comment).
The first two tests provide strong analytical oracles (no CPU
reference impl per feedback_no_cpu_test_fallbacks). The finite-diff
test cross-validates the full softmax+CE chain via a numerical
gradient — the standard ground-truth for autodiff kernels.
Forward-Return-Distribution head per SP20 §3 P3. Supervised forecaster
over 3 horizons × 21 return-bucket atoms — replaces the survivor-biased
checklist head per CRIT-1.
Architecture (2-layer MLP):
hidden [B, 64] = ReLU(h_t [B, 128] @ W1 [128, 64] + b1)
logits [B, 63] = hidden @ W2 [64, 63] + b2 // 63 = 3 × 21
Softmax + CE happen in the backward kernel (F.3). The forward kernel
caches the post-ReLU hidden buffer to avoid recomputing the W1 product
+ ReLU mask on backward.
Kernel `cuda/rl_frd_fwd.cu` — 1 block per batch, 64 threads:
* Phase 1 (tid < 64): each thread computes one hidden activation,
stages into shared mem, writes the cached `hidden_out[b, tid]`
* Phase 2 (tid < 63): each thread computes one output logit by
reading the shared hidden vector
* No atomicAdd (per-batch, per-output sole-writer pattern)
* No host branches in the launch (graph-capture safe)
Rust head module `src/rl/frd.rs`:
* `FrdHead::new(dev, cfg)` — Xavier × 0.1 init for W1/W2 (small enough
to keep initial softmax near-uniform), zero biases. Scoped-init-seed
guard per pearl_scoped_init_seed_for_reproducibility.
* `forward(h_t_d, hidden_out_d, logits_out_d, b_size)` — single
kernel launch via the cached `fwd_fn` handle.
* Public weight buffers (w1_d, b1_d, w2_d, b2_d) for the upcoming
bwd kernel + test harnesses.
* `pub const FRD_OUT_DIM = FRD_N_HORIZONS × FRD_N_ATOMS = 63` — single
canonical reference for the per-batch output width.
Tests `tests/frd_head.rs` — 3 GPU-oracle tests, all PASS on RTX 3050 Ti:
1. frd_forward_zero_input_emits_zero_logits — h_t=0 with default
b1=b2=0 must produce exactly zero logits AND zero cached hidden.
Unambiguous analytical oracle for the full matmul + ReLU + matmul
chain.
2. frd_forward_shape_matches_spec — random h_t produces correctly
shaped output [B × 63] with per-horizon softmax sums = 1.0
within 1e-5 (numerical-stable log-sum-exp).
3. frd_forward_relu_mask_consistent_with_cached_hidden — strictly
negative h_t input → ≥50% of cached hidden slots must be exactly
zero (ReLU fires). Empirically 128/256 zeros on the seeded init.
Per feedback_isv_for_adaptive_bounds: bucket-range σ stays in ISV
(slot 503, seeded ±3σ); only the 21-atom count is structural
compile-time per SP20 §0.1.
scripts/audit-wiring.sh dogfood pass flagged a7 (TrailTighten) and
a8 (TrailLoosen) as actions with no consumer anywhere in the
codebase (canonical pearl_dead_trail_stop_actions_a7_a8). Fix
bundles SP20 P1 (per-unit trade state buffers) and P5 (trail-stop
kernels) since they're the same architectural work.
Three new kernels:
rl_unit_state_update.cu — per-batch trade state machine. Runs
AFTER fill+extract_realized_pnl_delta.
Detects open/close/reverse position
transitions and populates unit slot 0
with entry_price, entry_step, lots,
initial_r, trail_distance. Slots 1-3
allocated for SP20 P7 pyramid expansion
but unused this commit.
rl_trail_mutate.cu — handles a7/a8 actions. Mutates ALL
active units' trail_distance bounded
by ISV [MIN, MAX] with symmetric
reciprocal adjust rate per SP20 §4.12:
a7: trail = max(MIN, trail × rate)
a8: trail = min(MAX, trail / rate)
rl_trail_stop_check.cu — per-batch per-unit breach check. Reads
shared lobsim best book (bid/ask),
computes mid, compares to each active
unit's (entry ± trail). On breach,
OVERRIDE actions[b] to FlatFromLong
(a3) or FlatFromShort (a4). Force-close
routes through existing flat plumbing
per pearl_stop_checks_run_at_deadline_cadence.
SP20 v3 §3 P5 calls for routing close
via partial-flat (a9/a10) so only the
at-risk unit closes — that needs P4
(N_ACTIONS=11). For now, ANY unit's
breach closes ENTIRE position via full
FlatFromLong/Short.
Per-batch per-unit buffers (8 new in trainer):
unit_entry_price_d [B × 4] f32
unit_entry_step_d [B × 4] i32
unit_lots_d [B × 4] i32
unit_initial_r_d [B × 4] f32
unit_trail_distance_d[B × 4] f32
unit_active_d [B × 4] u8
pyramid_units_count_d[B] i32
unit_prev_pos_lots_d [B] i32 (state-machine tracker, separate
from extract_realized_pnl_delta's
prev_position_lots_d for clean
kernel composability)
4 new ISV slots (494-497):
RL_TRAIL_MIN_INDEX — trail distance floor (seed 0.001)
RL_TRAIL_MAX_INDEX — trail distance ceiling (seed 100.0)
RL_TRAIL_K_INIT_INDEX — initial trail multiplier (seed 2.0, Turtle 2N)
RL_TRAIL_ADJUST_RATE_INDEX — tighten ratio (seed 0.9; symmetric reciprocal for loosen)
RL_SLOTS_END: 494 → 498.
LobSim exposes bid_px_d() + ask_px_d() public accessors. RlLobBackend
trait extended with the two accessors; the LobSimCuda impl wires
through.
Override stack ordering per SP20 §2.3:
1. rl_pi_action_kernel (sample)
2. rl_trail_mutate (a7/a8 → mutate, before stop check)
3. rl_trail_stop_check (per-unit breach → override action)
4. actions_to_market_targets (execute, including overridden flat)
5. step_fill_from_market_targets
6. extract_realized_pnl_delta
7. rl_unit_state_update (detect post-fill transitions)
Audit infrastructure refined as part of dogfooding:
* audit-isv allowlist extended for BOOK_LEVELS (structural book
depth) and ACTION_* prefix (enum-mirror constants — these are
structural API contracts matching src/rl/common.rs::Action positions)
* audit-wiring action-handler regex now matches BOTH literal
`action == <idx>` and `action == ACTION_<UPPER_SNAKE>` patterns,
and treats != as a handler too (a guard against the action is
valid wiring)
Both `audit-isv.sh` and `audit-wiring.sh` PASS cleanly with the
full manifest. audit-diag scheduled for first SP20 phase that adds
diag fields (this commit deliberately keeps diag exposure minimal
— full per-unit + trail diag blocks come with SP20 P13).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two architectural fixes from rljzl in-flight analysis (ultrathink
deep dive on actions a7/a8 + per-action calibration):
(1) λ_distill: static → controller-driven via Schulman bounded step
wwcsz showed Q→π KL EMA dropped 2.10 → 0.30 with λ=0.01, then
rljzl bumped to 0.05. Static λ is design intuition; KL is the
natural feedback signal:
if KL > target × 1.5 → λ *= 1.2 (Q not landing, pull harder)
if KL < target / 1.5 → λ /= 1.2 (Q absorbed, relax)
Bounds [MIN=0.001, MAX=1.0]. Target KL seeded 0.1 (slot 491).
New kernel `rl_q_distill_lambda_controller.cu`. Runs after the
distill kernel writes KL_EMA each step.
(2) REWARD_SCALE_MIN: hardcoded 1e-3 → ISV-driven 1e-4
wwcsz audit (mean_abs_pnl_ema mean=920, max=49437, p99=high):
the controller wanted scale ≈ 3.5e-4 when EMA spiked to 2871
but pegged at 1e-3, letting scaled rewards exceed unit support
and wasting C51 atom resolution on outliers. ISV slot 492
permits runtime re-tuning; default 1e-4 admits one more order
of magnitude before pegging. Per user-stated "floors and clamp
bounds" exemption — ISV-resident for tunability, not because
required.
Diag exposes q_distill_kl_target + reward_scale_min so the new
adaptation chains are observable.
Investigation (ultrathink): actions 7/8 (TrailTighten/TrailLoosen)
have ZERO consumers across the codebase. Spec'd as "ISV mutation"
in actions_to_market_targets.cu header but no slot, no mutation
kernel, no stop-check kernel. ~10% of wwcsz policy mass goes to
dead no-ops. Documented in
`pearl_dead_trail_stop_actions_a7_a8.md` — implementation
deferred to its own SP (per-batch trail_distance buffer +
mutation kernel + stop-check integration with LobSim apply_fill_to_pos
per `pearl-stop-checks-run-at-deadline-cadence`). N_ACTIONS=9
preserved; alternative refactor to 7 actions captured as
"Path B" in the pearl.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two coupled fixes addressing vj5f6 findings:
(1) WIN_clamp oscillation — sparse-aware EMA
vj5f6 showed WIN_clamp oscillating 1.0 ↔ 67.0 across 40k steps.
Root cause: the Wiener-α blend in rl_reward_clamp_controller
treated pos_max=0 as "no win this step ≡ win magnitude is zero,"
exponentially decaying the EMA toward 0 during dry-spell windows
(no closed winning trades). With α=0.4, ten dry steps decayed EMA
by 0.6^10 ≈ 0.006, collapsing WIN back to MIN_WIN=1.0 floor.
Fix: only update pos_max_ema AND clip_rate_ema AND MARGIN when
pos_max > 0. A dry step is "no signal," not "zero signal." The
EMA retains its last winning-period estimate; the controller
doesn't ratchet on stale data.
(2) Q→π distillation — couples Q's improved calibration to π
vj5f6 showed l_q dropping 100× (2.37 → 0.02) but reward economics
IDENTICAL to 8xwq8 (no C51 V_MAX lift). Per Option B, π drives
action selection but is trained by PPO surrogate using advantage
= returns - V. V regression doesn't benefit from C51 calibration,
so Q's improved knowledge stays trapped in the critic head.
Deep audit revealed a self-reinforcing defensive trap:
Q learned "big positions lose money" → π_target favors small
actions → π picks a3+a4 (tiny long / Hold) → position lots ≈ 0
→ rewards mostly 0 → V learns "everything is 0" → V_pred ≈ 0
→ advantage = returns - V_pred ≈ 0 → PPO gradient ≈ 0 → π
frozen at defensive attractor → loop. Trade count dropped 3×
(rdgzl 25k → 8xwq8/vj5f6 9k closes per 10k steps), win rate
inversely correlated with l_q (50% early → 22% late) because
only forced closes happen (stops = losses).
Fix: new rl_q_pi_distill_grad.cu computes
π_target = softmax(E_Q[s,*] / τ)
∂L/∂logits[a] = λ × (π_new(a) - π_target(a))
and ADDS this gradient to pi_grad_logits AFTER the PPO surrogate
backward. Couples Q's preferences directly into π's update without
going through advantage. λ=0.01 (small, PPO dominant), τ=1.0
(canonical Boltzmann). 3 new ISV slots (λ + τ + KL_ema diag).
Diag exposes c51_v_max/v_min, q_distill_lambda/temperature, and
q_distill_kl_ema so the adaptation + distillation loop is observable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rdgzl follow-up — chain hypothesis layer 2:
reward clamp lift unlocked V regression + PPO advantage (R/done
-$1.39 → -$0.48), but Q's distributional learning was structurally
capped at hardcoded V_MAX=1.0 in bellman_target_projection.cu —
any Bellman target > 1.0 categorically projected to atom 20 (top)
regardless of clamp. Even with WIN=3.8 clamp, Q never saw a +3.8
reward signal as distinct from a +1.0 reward signal.
This commit makes V_MIN/V_MAX ISV-driven with monotone-grow ratchet
coupled to the reward clamp. The C51 distribution support adapts
WITHOUT destabilising Q's learned values — atom 20 always represents
at least the widest WIN we've ever admitted (only grows, never shrinks).
Implementation:
- 2 new ISV slots (484 V_MAX, 485 V_MIN) with [-1, +1] floors
seeded by rl_isv_write
- rl_reward_clamp_controller.cu also ratchets these slots:
V_MAX_new = max(V_MAX_prev, max(1.0, WIN_clamp))
V_MIN_new = min(V_MIN_prev, min(-1.0, -LOSS_clamp))
- bellman_target_projection.cu reads V_MIN/V_MAX from ISV, derives
DELTA_Z inline (was #define)
- New rl_atom_support_update.cu (21-thread block) refreshes
atom_supports_d = linspace(V_MIN, V_MAX, 21) per step so
downstream C51 kernels (argmax_expected_q, rl_action_kernel,
dqn_distributional_q) see the current span
- Trainer launches atom-support updater after each reward-clamp
controller launch (both helper + step_with_lobsim inline paths)
- Diag exposes c51_v_max + c51_v_min for adaptation visibility
Floors at [-1, +1] preserve original C51 design as hard minimum —
the atom support can only become wider, never narrower than the
baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
alpha-rl-rmgm5 (commit a776fab31) deep-diag finding:
- static `[-3, +1]` clamp fired on 85% of steps
- pre-clamp max: p95=15.5 p99=45.2 max=2830 (in WIN-bound units)
- win distribution avg=+$2.06 max=+$11 squished to +1.0
- loss distribution avg=-$3.84 routinely exceeded -3.0
- per-trade EV = 0.357 * 2.06 + 0.643 * (-3.84) = -$1.74
The static clamp was crushing the gradient differential between
profitable and unprofitable trades, leaving Q with no signal to
distinguish good actions from bad. Adaptive bounds let the actual
winning-trade distribution reach the C51 atom support.
Implementation:
- apply_reward_scale.cu: dual reduction (max|scaled| + max(positive
scaled, 0)); positive-tail published to new ISV slot 478
- rl_reward_clamp_controller.cu: maintains EMA of slot 478 in slot
479 via Wiener-α blend (floor 0.4 per pearl_wiener_alpha_floor);
writes WIN_eff = clamp(MARGIN * EMA, [1.0, 20.0]) to slot 452
and LOSS_eff = RATIO * WIN to slot 453
- 4 new ISV slots (478-481): raw + EMA + margin + ratio
- Trainer per-step launch added at both apply_reward_scale sites
(helper method + step_with_lobsim inline path)
- Shared-mem bytes doubled at both apply_reward_scale launches
- Static-default seeds added to with_controllers_bootstrapped
(MARGIN=1.5, RATIO=3.0) — controller's bootstrap-on-sentinel
path takes over from these once first positive reward observed
- Diag JSONL exposes pos_scaled_max, pos_scaled_max_ema, and the
margin/ratio config
Preserves 3:1 loss-aversion asymmetry per
pearl_audit_unboundedness_for_implicit_asymmetry — RATIO is itself
ISV-tunable. WIN floor 1.0 / ceiling 20.0 are hardcoded per the
user-stated "floors and clamp bounds" exemption (2026-05-24).
Adds #![recursion_limit = "256"] to alpha_rl_train.rs — the diag
json! block crossed serde_json's default 128-arg expansion budget.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two coordinated architectural fixes addressing the deepest blockers
exposed by the audit:
## Option B: π-driven action selection
Per `pearl_q_thompson_actor_makes_pi_dead_weight`: the prior
architecture had Q acting as BOTH actor (via Thompson sample) AND
critic (via Bellman target). π trained by PPO surrogate against
Q's actions but never drove any decision — `q_pi_agree_ema`
decayed to 0 by step 5000 in every smoke because π converged to
Q's Thompson SAMPLING distribution, not Q's argmax. π was
dead-weight: 4 dedicated controllers (ε, ratio_clamp,
entropy_coef, KL EMA), shared encoder gradient interference, and
zero contribution to actor decisions.
### New kernel: rl_pi_action_kernel.cu
Single-thread-per-batch CUDA kernel that:
1. Computes numerically-stable softmax(pi_logits[b, :])
2. Draws u ∈ [0, 1) from per-batch xorshift32 PRNG
3. CDF-walks to pick the multinomial-sampled action
Per-batch xorshift32 PRNG state is the SAME `prng_state_d` buffer
already used by rl_action_kernel — no new state needed. Sampling
deterministic given (seed, b_size, pi_logits).
### Trainer wiring (1 site change in step_with_lobsim)
Replaced `rl_action_kernel(q_logits, atom_supports, ...)`
(Q-Thompson) with `rl_pi_action_kernel(pi_logits, ...)`
(π-multinomial). The argmax_expected_q call on h_{t+1} is
unchanged — Q remains the critic via canonical Double-DQN target.
PPO importance-ratio surrogate now has its canonical actor-critic
semantics: π_new(a|s) / π_old(a|s) where `a` was actually sampled
from π_old. Was nonsensical before (a was sampled from Q-Thompson,
not π, so the ratio measured something incoherent).
The rl_action_kernel (Q-Thompson) cubin + function field are kept
loaded for backward-compat tests and diagnostic comparison; no
longer in the hot path.
## b_size: 1 → 16
Per `pearl_b_size_1_signal_starvation_blocks_q_learning`: at
b_size=1 with 11% done-step rate and 70% loss rate per trade, Q
stayed at uniform baseline ln(21)=3.04 across all 16+ smokes
regardless of controller fixes. The architecture was structurally
signal-starved — 1 gradient sample per Adam step is fundamentally
too noisy.
LobSimCuda already supports b_size>1 (n_backtests parameter at
`crates/ml-backtesting/src/sim/mod.rs:355`). Trainer code is
already b_size-parametric throughout. The blocker was just the
CLI default at `--n-backtests=1`.
Default bumped to 16 (matches the doc note "production sweep at
32-64; L40S 48GB"). 16× more gradient samples per Adam step
gives Q proper batch variance reduction. The K-loop multiplier
(`isv[404]/2048`) will likely settle at K=1 since the
advantage_var_ratio drops with batch size.
## Expected behaviour
* `q_pi_agree_ema` becomes tautological/dropped (π IS the
policy now — comparing argmax(Q) to argmax(π) doesn't measure
a real consistency invariant any more)
* π gradient flows naturally drive π toward an actor that
optimises the PPO surrogate — Q's encoder gradient is no
longer competing with a different policy's gradient
* l_q should drop meaningfully below 3.04 for the first time
(was stuck at 2.7-2.9 across all prior smokes)
* reward/trade should approach 0 (was -$0.5 to -$0.8 across
every prior run)
* Wall-clock per env step ~16× slower (b_size=16) but training
cost per gradient step similar (denser sample = more
progress per step)
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
## Caveat: integrated_trainer_smoke runs at b_size=1
The default for the CLI is bumped to 16, but the local
`integrated_trainer_smoke` test passes its own b_size=1 to
verify the trainer mechanics. Real-world signal verification
happens via cluster smokes which now use b_size=16 by default.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per `feedback_isv_for_adaptive_bounds`: every controller design knob
that's genuinely tunable now lives in ISV instead of as a kernel-side
`#define`. Tuning is a re-seed (kernel launch with new arg) rather
than a recompile.
## New ISV slots (10 design constants)
RL_REWARD_CLAMP_WIN_INDEX (452, =1.0) apply_reward_scale
RL_REWARD_CLAMP_LOSS_INDEX (453, =3.0) apply_reward_scale
RL_KL_TARGET_INDEX (454, =0.01) rl_ppo_clip_controller
RL_IMPROVEMENT_THRESHOLD_INDEX (455, =0.99) rl_lr_controller
RL_PLATEAU_PATIENCE_INDEX (456, =1000.0) rl_lr_controller
RL_DIV_TARGET_INDEX (457, =0.01) rl_target_tau_controller
RL_ENTROPY_TARGET_FRAC_INDEX (458, =0.7) rl_entropy_coef_controller
RL_KURT_LIFT_SCALE_INDEX (459, =7.0) rl_per_alpha_controller
RL_PPO_CLAMP_MARGIN_INDEX (460, =10.0) rl_ppo_ratio_clamp_controller
RL_LR_WARMUP_STEPS_INDEX (461, =2000.0) rl_lr_controller
RL_SLOTS_END: 452 → 462.
## Constants NOT converted (truly fundamental)
* All `*_INDEX` (ABI)
* All `*_MIN`/`*_MAX` clamp bounds (algebraic domain)
* All `*_BOOTSTRAP` (one-shot init)
* `WIENER_ALPHA_FLOOR` (per pearl_wiener_alpha_floor_for_nonstationary)
* Schulman pattern parameters (`*_TOLERANCE`/`*_ADJUST_RATE`)
* C51 (`Q_N_ATOMS`, `V_MIN/MAX`, `N_ACTIONS`)
* Kernel numerics (`STREAM_ALPHA`, `ABS_MEAN_FLOOR`, `EPS_PNL`)
* `KURT_GAUSSIAN` (statistical constant = 3.0 for Gaussian)
* `KURT_NOISE_FLOOR` (defensive)
* `LR_BOOTSTRAP`/`LR_MIN`/`LR_MAX`/`LR_LOSS_EMA_ALPHA`/`DECAY_FACTOR`
## New infrastructure
New CUDA kernel `rl_isv_write.cu` — generic single-thread device-side
seeder taking `(int slot, float value)`. Trainer loops calling it
once per design constant at init. Replaces the prior pattern of
extending `rl_streaming_clamp_init`'s arg list every time a new
constant was added.
## Ordering fix
Design constants must be seeded BEFORE controllers bootstrap — the
controllers' bootstrap paths read these slots (e.g.
`rl_entropy_coef_controller` reads `RL_ENTROPY_TARGET_FRAC_INDEX`
to derive its target). Without correct ordering, controllers see
sentinel 0.0 and bootstrap to wrong values (caught by failing G1
test before commit). Seed loop runs at TOP of
`with_controllers_bootstrapped`.
## Diag bake-in
JSONL gains `isv_config` block exposing all 10 design constants per
step:
isv_config.{reward_clamp_win, reward_clamp_loss, kl_target,
improvement_threshold, plateau_patience, div_target,
entropy_target_frac, kurt_lift_scale, ppo_clamp_margin,
lr_warmup_steps}
Post-hoc analysis can correlate any controller's behaviour with the
exact design constants it saw, without grepping the source for
`#define` defaults.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) — skip 10 new design-
constant slots in sentinel-zero loop, assert seeded values
separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with 10 new assertions)
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
gxhr8 confirmed the streaming kernels work — both formerly-dead
controllers (rl_rollout_steps, rl_per_alpha) now adapt instead of
pegging at MIN. But the unclamped streaming outputs reached
advantage_var_ratio = 3e5 (when streaming-mean passed through zero
and `var/|mean|` blew up under the 1e-6 denominator floor) and
td_kurtosis = 50.6, pegging both downstream controllers at MAX
instead. Per_α at MAX over-concentrates PER sampling on outliers,
which hurts distributional Q learning (best l_q window regressed
from 2.41 → 2.69 between pdgxn and gxhr8).
## Fix: ISV-resident output clamp ceilings
Two new ISV slots hold the streaming-kernel output ceilings:
RL_ADV_VAR_RATIO_CLAMP_INDEX = 447 (default 100.0)
RL_TD_KURTOSIS_CLAMP_INDEX = 448 (default 30.0)
* 100.0 for var_ratio = 1000× ADV_VAR_RATIO_TARGET (= 0.1) — wide
enough that healthy signal (typical 1-10) passes through, tight
enough that 3e5 outliers don't peg rollout_steps.
* 30.0 for kurtosis = 3× (KURT_GAUSSIAN + KURT_LIFT_SCALE) — lets
the full per_α response range engage on heavy-tailed signal
(≤ 10), bounds runaway above that.
Per `feedback_isv_for_adaptive_bounds`: the clamps live in ISV
(visible in diag, modifiable at runtime via re-launching the init
kernel or a future adaptive controller) rather than as kernel-side
`#define`s.
## Seeding (no HtoD per feedback_no_htod_htoh_only_mapped_pinned)
New device kernel `rl_streaming_clamp_init.cu` — single thread,
writes both clamp ceilings directly to ISV. Launched once at the
end of `with_controllers_bootstrapped` alongside the 8 existing
controller-bootstrap launches. Zero host→device transfer.
## Diag bake-in (per user request "ensure to bake in diags")
JSONL gains a new `streaming` block exposing:
* `streaming.adv_var.{mean, m2, clamp}`
* `streaming.td_kurt.{mean, m2, m4, clamp}`
Cross-check: when consumer-input slot (RL_ADVANTAGE_VAR_RATIO_EMA_INDEX
or RL_TD_KURTOSIS_EMA_INDEX) reads exactly the same value as
`streaming.*.clamp`, the clamp fired this step.
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert that
ISV[417..END] is sentinel-zero at bootstrap. Both new slots are
seeded to non-zero values by rl_streaming_clamp_init during
bootstrap, so both tests skip these slots in the loop and assert
the seeded values separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mjzfk + pdgxn diags showed `advantage_var_ratio` and `td_kurtosis`
identically 0 for 100% of every 50k-step smoke. Root cause: the
per-batch `rl_var_over_abs_mean_b` and `rl_kurtosis_b` kernels are
mathematically undefined at b_size=1 (variance of a single sample is
zero; kurtosis of a single sample is 0/0). The kernels correctly
returned 0 in that case but the downstream `rl_rollout_steps` and
`rl_per_alpha` controllers then never saw signal and pegged at MIN
(2048 / 0.4) for the entire run.
## Fix: time-axis Welford-EMA streaming
Replace per-batch reduction with per-step EMA-streaming moments
maintained in ISV slots:
rl_var_over_abs_mean_streaming.cu — maintains streaming mean + M2,
emits var/|mean| each step. Welford-EMA on the batch-mean of
advantages_d (one value at b_size=1, or a single batch reduction
at b_size>1) folded into the time-axis estimator.
rl_kurtosis_streaming.cu — maintains streaming mean + M2 + M4,
emits M4/M2² (Pearson kurtosis) each step. Same Welford-EMA shape
applied to td_per_sample_d batch mean.
Both kernels use STREAM_ALPHA = 0.05 (matches LR_LOSS_EMA_ALPHA —
half-life ≈ 14 steps) so the time estimator smooths over noisy
per-step batch-mean observations. The kernel writes the smoothed
estimate DIRECTLY to the controller-input ISV slot
(RL_ADVANTAGE_VAR_RATIO_EMA_INDEX = 421,
RL_TD_KURTOSIS_EMA_INDEX = 422); the prior downstream
ema_update_per_step calls for these two signals are REMOVED — the
streaming kernel IS the EMA.
## ISV slot allocation
5 new state slots holding the streaming-mean / M2 / M4 per-stream
state. RL_SLOTS_END: 442 → 447.
RL_ADV_VAR_STREAM_MEAN_INDEX = 442 (streaming mean of advantages)
RL_ADV_VAR_STREAM_M2_INDEX = 443 (streaming M2 of advantages)
RL_TD_KURT_STREAM_MEAN_INDEX = 444 (streaming mean of TD-CE)
RL_TD_KURT_STREAM_M2_INDEX = 445
RL_TD_KURT_STREAM_M4_INDEX = 446
Per `pearl_first_observation_bootstrap`: sentinel-zero state
triggers replace-direct first-observation bootstrap (the first
step seeds μ = batch_mean, M2 = 0, M4 = 0 — subsequent steps blend).
Per `pearl_blend_formulas_must_have_permanent_floor`: var/|mean|
denominator floored at 1e-6, M2² denominator floored at 1e-12 —
prevents div-by-zero blow-up when streaming mean / variance is
genuinely zero (cold-start or quiet regime).
## Files
* crates/ml-alpha/cuda/rl_var_over_abs_mean_streaming.cu — new
* crates/ml-alpha/cuda/rl_kurtosis_streaming.cu — new
* crates/ml-alpha/cuda/rl_var_over_abs_mean_b.cu — deleted
* crates/ml-alpha/cuda/rl_kurtosis_b.cu — deleted
* crates/ml-alpha/src/rl/isv_slots.rs — +5 slots
* crates/ml-alpha/src/trainer/integrated.rs — rewired
launchers,
dropped
redundant
ema_update
calls
* crates/ml-alpha/build.rs — swapped
cubin
entries
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers ✅
G4 target_update ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
pt67l confirmed reward-scale + V-target clamp eliminate V regression
spikes — but exposed a residual: |l_pi| max=586 with mean 0.22. Root
cause: PPO's clip(r, 1-ε, 1+ε) bounds the loss only when surr2 is
the active min. The unclipped branch IS active when A<0,r>1+ε
(surr1=A·r is then more negative than surr2=A·(1+ε), so min selects
surr1) and when A>0,r<1-ε. In the first case `r` can blow up: we've
seen r reach 1e10 from policy drift over a multi-step rollout
producing l_pi=O(1e10) spikes that contaminate the loss-balance
controller and the LR controller's plateau detection.
## Fix: ISV-driven ratio clamp
Per `feedback_isv_for_adaptive_bounds` and
`pearl_controller_anchors_isv_driven`: the clamp ceiling lives in
ISV[RL_PPO_RATIO_CLAMP_MAX_INDEX = 440], not as a hardcoded #define.
New controller `rl_ppo_ratio_clamp_controller.cu`:
* Anchors on the (already KL-adaptive) PPO clip ε at ISV[402]
* target = (1 + ε) × PPO_CLAMP_MARGIN (MARGIN = 10.0)
* Wiener-α blend with floor 0.4 per
pearl_wiener_alpha_floor_for_nonstationary (ε is non-stationary)
* Permanent floor 2.0 / ceiling 1000 per
pearl_blend_formulas_must_have_permanent_floor
* Bootstrap 10.0, replace-directly on first non-bootstrap ε
observation per pearl_first_observation_bootstrap
When ε is small (rl_ppo_clip_controller seeing low KL → tight clip
band), the ratio clamp tightens — outliers should be rare anomalies.
When ε widens (large KL → wide clip band), the clamp widens
proportionally — outliers are expected so we permit more
magnitude before bounding.
## Wiring
ppo_clipped_surrogate_fwd and _bwd both read
isv[RL_PPO_RATIO_CLAMP_MAX_INDEX] and clamp ratio to
[1/ratio_max, ratio_max] before forming surr1/surr2. The clamp is
forward-only in effect (bwd gates pg_grad inside [1-ε, 1+ε] anyway
so gradients were already bounded), but bounding the FORWARD ratio
keeps l_pi sane for the controllers downstream.
The new controller is wired into both:
* `with_controllers_bootstrapped` — bootstrap launch alongside
the other 7 R1 controllers
* `launch_rl_controllers_per_step` — per-step refresh alongside
the other 7 R5 controllers
## Diagnostic: per-step max |log_ratio|
New kernel `ppo_log_ratio_abs_max_b.cu` (same tree-reduce shape as
rl_kl_approx_b) writes per-batch max(|log π_new − log π_old|) to
ISV[RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441]. Launched right after
rl_kl_approx_b (uses the same log_pi_old_d + pi_log_prob_d inputs).
Surfaces in diag JSONL as:
"ppo": {
"ratio_clamp_max": isv[440], # adaptive ceiling
"log_ratio_abs_max": isv[441] # per-step observed max
}
The clamp fires when log_ratio_abs_max > ln(ratio_clamp_max).
For ratio_clamp_max = 10, ln = 2.30. Healthy training has
log_ratio_abs_max well below this most steps; outliers touch or
exceed it on rare excursions which the clamp bounds before they
pollute l_pi.
## Slot allocation
RL_PPO_RATIO_CLAMP_MAX_INDEX = 440 (controller output)
RL_PPO_LOG_RATIO_ABS_MAX_INDEX = 441 (per-step diag)
RL_SLOTS_END = 442 (was 440)
## Test updates
G1 (isv_bootstrap) + G3 (r5_controllers) blanket-assert ISV[417..END]
== 0.0 to catch slot-wiring bugs. Slot 440 is now a controller
OUTPUT bootstrapped to 10.0, so both tests skip it in the loop and
assert == 10.0 separately.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (with new slot-440 assertion)
G3 controllers ✅
G4 target_update ✅
G6 r7d_per_wiring ✅
integrated_smoke ✅
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The rl_lr_controller emitted a hardcoded `LR_BOOTSTRAP = 1e-3` for
every step regardless of training dynamics. The kernel accepted 5
`*_signal` scalar args but ignored them via `(void)signal;` — a
stub. This commit makes the LR genuinely signal-driven per
`pearl_controller_anchors_isv_driven` + `feedback_isv_for_adaptive_bounds`.
## Architecture
Per-head grad-norm EMA → LR target derivation:
observed_grad_norm = EMA(‖grad_w_head‖₂)
target_lr = lr_prev × (TARGET_GRAD_NORM / max(observed, ε))
Wiener-α blend (floor 0.4) + clamp to [LR_MIN, LR_MAX].
Multiplicative pattern — same shape as rl_target_tau / rl_ppo_clip
controllers. High observed gradient (model thrashing) shrinks LR
(calm updates); low observed gradient (model coasting) grows LR
(push more aggressive learning).
## Components
1. **rl_l2_norm.cu** (new) — single-buffer L2 norm `‖x‖₂` via
grid-stride loop + shared-mem tree reduce. Used for per-head
grad_w_*_d reductions.
2. **rl_lr_controller.cu** (rewrite) — kernel signature changes
from 5 scalar `*_signal` args to 5 `int *_signal_slot` args
(ISV slot indices). The kernel reads each signal from
`isv[slot]`, derives target multiplicatively, and applies the
cold-start gate + replace-directly pattern (same R9-audit fixes
that closed the dead-zones in the other multiplicative
controllers). BCE and AUX heads pass sentinel `-1` for their
signal slot (those heads are owned by the perception trainer);
the kernel falls back to LR_BOOTSTRAP for those.
3. **ISV slot extension** (`isv_slots.rs`):
* `RL_Q_GRAD_NORM_EMA_INDEX = 424`
* `RL_PI_GRAD_NORM_EMA_INDEX = 425`
* `RL_V_GRAD_NORM_EMA_INDEX = 426`
* `RL_SLOTS_END = 427` (was 424).
4. **Trainer wiring** (`integrated.rs`):
* New `rl_l2_norm` module + fn fields + load in `new()`.
* New `launch_l2_norm` helper (256-thread single-block reduce).
* After-encoder-backward block in `step_synthetic` gains 3
grad-norm + EMA launches (Q grad_w 24,192 floats, π grad_w
1,152 floats, V grad_w 128 floats) alongside the existing
entropy / td_kurtosis / kl_pi EMAs.
* `launch_rl_lr_controller` updated to pass i32 slot indices
instead of f32 scalars.
## What's NOT in this commit
* BCE and AUX LR signals — those heads' gradients live in the
perception trainer, not the RL trainer. A future commit can
wire `perception.bce_grad_w_d` → ISV slot if the BCE/AUX LRs
need to adapt for cross-trainer alignment.
* Production tuning of `TARGET_GRAD_NORM = 1.0`. Empirical from
the 50k smoke (Q grad_w L2 norm landed near 1 at LR=1e-3); the
smoke at this commit will confirm whether the LR controller
drives the grad-norm to this anchor.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅ (per_α, γ, etc. — unchanged)
G3 controllers_emit ✅ (test pre-seeds inputs)
G4 target_soft_update ✅
G6 r7d_per_wiring ✅
smoke ✅ all losses finite
## Expected effect
Prior 50k run showed l_q oscillating in 2.7-4.4 range without
visible convergence at LR=1e-3 constant. With LR now adaptive,
the trainer should:
* Shrink Q LR when Q grad-norm spikes (large per-sample CE
after a big trade close).
* Grow Q LR when grad-norm stays small (steady-state coasting).
* Same logic for π and V.
Next cluster smoke at 50k steps will produce a diag.jsonl where
ISV[413..415] (lr_q, lr_pi, lr_v) AND ISV[424..426] (grad-norm
EMAs) both evolve over time — observable convergence dynamics
that previously didn't exist.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the controller-input EMA wiring across all 7 RL
controllers. Previously 6 of 7 EMA input slots received no signal,
freezing those controllers at bootstrap for the entire training run.
Three new derivation kernels populate the last three:
* `rl_kl_approx_b` — Schulman-style per-step KL approximation
`mean(log π_old(a) − log π_new(a))` over batches at the sampled
action. Single-block tree reduce; inputs are `log_pi_old_d`
(recorded at action sample time) and `pi_log_prob_d` (= log
π_new at the same action, output of PPO surrogate forward).
Feeds `kl_pi_ema` (ISV[419]) consumed by rl_ppo_clip.
* `rl_l2_diff_norm` — `‖W_online − W_target‖₂` over the full
DQN weight tensor (24,192 floats = 9 actions × 21 atoms × 128
hidden). Single-block grid-stride loop (256 threads, ~95
iterations each); shared-mem tree-reduce produces a scalar.
Feeds `q_divergence_ema` (ISV[418]) consumed by rl_target_tau.
Launched immediately after `soft_update_target` so the divergence
reflects the post-update gap.
* `rl_step_counter_update` — per-batch trade-duration counter +
done-gated emit. Trainer-owned `steps_since_done_d: [i32; B]`
increments every step and resets on done; on done the counter
value (= event count the position was open) is written to
`trade_duration_emit_d` for `ema_update_on_done` to fold into
`mean_trade_duration_ema` (ISV[417]) consumed by rl_gamma.
Element-wise, one thread per batch index.
## Trainer wiring placement
* trade_duration counter + EMA emit: in `step_with_lobsim`
immediately after `extract_realized_pnl_delta` populates dones_d,
BEFORE the controllers fire — γ adapts THIS step from a real
duration observation.
* q_divergence reduce + EMA: in `step_with_lobsim` immediately
after `dqn_head.soft_update_target` runs, so the divergence
captures the post-update gap. One step lag for the τ controller
(controllers fired earlier in step_with_lobsim, before
step_synthetic).
* kl_pi reduce + EMA: in `step_synthetic` end-of-function block
alongside entropy + td_kurtosis updates. Deferred to AFTER the
encoder backward so the `&self.perception` borrow held by
`h_t_borrow` releases before the `&mut self` launches. One
step lag for the ε controller.
## All 7 controller inputs now wired
| ISV slot | Input EMA | Producer |
|----------|-----------|----------|
| 417 | mean_trade_duration | rl_step_counter_update → ema_update_on_done |
| 418 | q_divergence | rl_l2_diff_norm → ema_update_per_step |
| 419 | kl_pi | rl_kl_approx_b → ema_update_per_step |
| 420 | entropy_observed | ema_update_per_step (direct mean on entropy_d) |
| 421 | advantage_var_ratio | rl_var_over_abs_mean_b → ema_update_per_step |
| 422 | td_kurtosis | rl_kurtosis_b → ema_update_per_step |
| 423 | mean_abs_pnl | abs_copy + ema_update_on_done (existing) |
Combined with the cold-start gate + replace-directly-on-first-warm
fixes from the prior two commits, all 7 controllers will:
1. Hold at bootstrap until their input EMA receives signal
(cold-start gate prevents migration to clamps during sentinel
input period).
2. Replace prev → target directly on first non-zero observation
(no 60% bootstrap contamination in the first warm step).
3. Wiener-α blend (floored at 0.4) on subsequent steps.
## Phase-prefix comment cleanup
Per directive to stop phase prefixing in code, scrubbed "R9 audit",
"Phase A", "Phase B" markers from comments I added across the
multi-commit fix sequence. The remaining "Phase B: cross-batch
param-grad reducer" in build.rs is a pre-existing comment on the
perception trainer's `reduce_axis0` kernel, unrelated to this work.
## Verified gates (local sm_86)
G1 isv_bootstrap ✅
G3 controllers_emit ✅ (test pre-seeds inputs)
G4 target_soft_update ✅
G6 r7d_per_wiring ✅
R3, R4, smoke ✅
The next cluster smoke at this SHA will produce a diag.jsonl where
ALL 7 controller-input EMAs evolve over the 1000 steps, and ALL 7
controllers visibly adapt — the first time the integrated trainer
has every adaptive controller wired since the rebuild plan was
written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
R9 cluster smoke alpha-rl-qzstj diag exposed that 6 of 7 controllers
held at bootstrap for the entire 1000-step run because their input
EMAs were never populated. Only `mean_abs_pnl_ema` was wired (via
ema_update_on_done on reward_abs_d). The other 6 EMA producers
existed as generic kernels (ema_update_per_step / ema_update_on_done)
but nothing computed the per-step input signals to feed them.
This commit wires the 3 EMAs whose source signals are ALREADY
computed and live in trainer per-step buffers (Phase A — cheapest
to wire):
* `entropy_observed_ema` (ISV[420] → rl_entropy_coef controller)
← per-batch entropy `entropy_d` from PPO surrogate forward.
`ema_update_per_step` does mean-reduce internally, so this is
a single launch with entropy_d as input (b_size native).
* `advantage_var_ratio_ema` (ISV[421] → rl_rollout_steps)
← `var(advantages) / max(|mean(advantages)|, 1e-6)` reduction
on advantages_d. New kernel `rl_var_over_abs_mean_b`
(two-pass shared-mem tree-reduce) writes scalar to trainer-
owned `ema_input_scratch_d[1]`, then `ema_update_per_step`
consumes with b_size=1.
* `td_kurtosis_ema` (ISV[422] → rl_per_alpha)
← `E[(x-μ)⁴] / σ⁴` kurtosis reduction on td_per_sample_d
(R7d's per-sample CE loss from dqn_distributional_q_bwd).
New kernel `rl_kurtosis_b` (three-pass shared-mem tree-reduce)
writes scalar to ema_input_scratch_d, then ema_update_per_step
with b_size=1.
## Wiring placement
* `advantage_var_ratio` update: in `step_with_lobsim` immediately
after `compute_advantage_return` populates `advantages_d`. Fires
BEFORE the next step's controllers, so the controller sees the
fresh signal one step later.
* `entropy_observed` + `td_kurtosis` updates: in `step_synthetic`
AFTER the encoder backward (deferred from their natural in-place
locations to avoid a borrow-checker conflict with `h_t_borrow`
which holds `&self.perception` through the entire forward chain).
One-step lag — same as advantage_var_ratio for the same reason
(controllers fire in the NEXT step_with_lobsim).
## Trainer-owned scratch
Single `ema_input_scratch_d: CudaSlice<f32>` of length 1. Reused
across the var-over-abs-mean and kurtosis launches in any given
step — they're stream-serialised, so the second reducer's write
to slot 0 strictly follows the first reducer's consumer (the
corresponding ema_update_per_step). Cheap (4 bytes); avoids
two separate scratches.
## Why a 1-float scratch + b_size=1 ema_update
`ema_update_per_step` expects `obs_d[b_size]` and computes per-step
mean as `Σobs / b_size`. Passing a 1-element buffer gives
mean = obs[0] = the reduce kernel's scalar output. The EMA then
blends `prev` toward that scalar via Wiener-α (or bootstraps on
first non-zero per `pearl_first_observation_bootstrap`).
This pattern lets the existing per-step EMA kernel handle scalar
inputs without modification — the alternative (a dedicated
"ema_scalar_per_step") would duplicate logic per
`feedback_single_source_of_truth_no_duplicates`.
## Verified gates (post-fix, local sm_86)
G1 isv_bootstrap ✅
G3 controllers_emit ✅ (test pre-seeds inputs, so
wiring path not exercised)
G4 target_soft_update ✅
G6 r7d_per_wiring ✅
R3, R4, smoke ✅
Local smoke at b_size=1 won't exercise kurtosis (kernel returns 0
at b_size<2 → cold-start gate holds per_α at bootstrap). var_over_
abs_mean does fire because b_size=1 has a well-defined (degenerate)
variance of 0. Cluster smoke at b_size=1 will mostly exercise
entropy_observed.
## What's NOT in this commit (Phase B — 3 EMAs left)
* `kl_pi_ema` (ISV[419] → rl_ppo_clip)
needs: D_KL approximation between log_pi_old and log_pi_new.
Both buffers exist in trainer; need a small subtract-and-mean
kernel OR extend PPO surrogate forward to emit kl_per_batch.
* `q_divergence_ema` (ISV[418] → rl_target_tau)
needs: `‖W_online − W_target‖₂`. Both DQN weight buffers
accessible via dqn_head fields; need a small L2-diff-norm
kernel called after soft_update_target.
* `trade_duration_ema` (ISV[417] → rl_gamma)
needs: per-batch step counter (i32, b_size, trainer-owned)
that increments each step and emits its value on done. Needs
a small `step_counter_update` kernel + the counter buffer.
These three need NEW signal-derivation kernels (not just reductions
over existing buffers). Separate commit.
## Cluster smoke expected diag change
Before this commit: 6 of 7 EMA input slots stuck at 0.0 for all
1000 steps. After: ISV[420], ISV[421], ISV[422] populated each
step. The corresponding controllers (coef, n_roll, per_α) should
visibly adapt after the first few non-zero observations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes plan A9 (rebuild plan's "PER wiring" R7 scope second half;
R7c-data shipped the first half last commit). The `ReplayBuffer` in
`src/rl/replay.rs` has sat as dead code since Phase C — this commit
makes it load-bearing per `feedback_always_per` ("PER always enabled;
non-PER paths are dead code").
## Architecture: off-policy Q + on-policy PPO + V + stop-grad encoder
Shared-encoder pattern with the canonical off-policy + shared-encoder
discipline: the Q head trains from PER-sampled past transitions,
PPO + V train on current-step on-policy data, and the encoder receives
gradient signal ONLY from PPO + V (and BCE/aux via the perception
trainer's separate `step_batched` path). Standard pattern in SAC,
R2D2, IMPALA.
Stop-grad is implemented by computing Q's `grad_h_t` (via
`backward_to_w_b_h(sampled_h_t, ...)`) but NOT accumulating it into
`grad_h_t_combined_d` — the encoder backward only sees π + V
contributions. Per `feedback_no_hiding` the discarded buffer is
allocated and written (the kernel API requires the writeback target);
the discard is a deliberate design call documented at the
accumulation site.
## Wiring summary
### Kernel: `dqn_distributional_q_bwd`
* New `loss_per_batch [B]` output. Atom 0 of each block writes the
per-sample CE loss (non-atomic — single writer per batch).
`loss_out [1]` continues to atomicAdd the scalar sum for the
diagnostic total. Build.rs cache bust v30.
### `DqnHead::backward_logits` (Rust wrapper)
* New `loss_per_batch: &mut CudaSlice<f32>` arg. Migrated atomically
in the same commit per `feedback_no_partial_refactor` — only
caller is the integrated trainer.
### `IntegratedTrainerConfig`
* New `per_capacity: usize` (default 4096, matches `replay.rs` doc
ceiling for naive O(N) sampling).
* New `per_seed: u64` (default 0x9E37_79B9_7F4A_7C15).
* `Default` impl added so test fixtures forward-compat via
`..IntegratedTrainerConfig::default()`. All 5 existing test
fixtures migrated.
### `IntegratedTrainer`
* New fields: `replay: ReplayBuffer`, `sampled_h_t_d`,
`sampled_h_tp1_d`, `sampled_actions_d`, `sampled_rewards_d`,
`sampled_dones_d`, `sampled_next_actions_d`, `td_per_sample_d`.
* New methods: `push_to_replay(b_size)` — DtoH per-batch metadata
(action/reward/done/log_pi_old) + alloc per-transition
`CudaSlice<f32>(HIDDEN_DIM)` ×2 + DtoD per-batch slice copies +
push to `ReplayBuffer`. `sample_and_gather(b_size)` — read
per_α from ISV[405], call `replay.sample_indices`, gather sampled
transitions' h_t/h_tp1 device payloads via per-batch DtoD into
`sampled_h_t_d` / `sampled_h_tp1_d`, HtoD upload action/reward/done.
### `step_with_lobsim` orchestration
After `compute_advantage_return` and BEFORE `step_synthetic`:
1. DtoH full ISV slice to refresh `isv_host` (so PER reads ISV[405]
for per_α).
2. `push_to_replay(b_size)` — push current step's transitions.
3. `sample_and_gather(b_size)` — return `per_indices` for the
priority update.
4. `step_synthetic(snapshots)` — runs π + V on current-step h_t,
Q on SAMPLED h_t (off-policy).
5. DtoH `td_per_sample_d` → host; `replay.update_priorities(
per_indices, td_per_sample_host)`.
6. Target-net soft update (unchanged from R5).
### `step_synthetic` redirects (Q path → sampled, π/V stay on-policy)
* Q forward: `forward(&self.sampled_h_t_d)` (was `h_t_borrow`).
* New: forward online Q on `&self.sampled_h_tp1_d` → local scratch +
`argmax_expected_q` → `self.sampled_next_actions_d`. The
Double-DQN argmax MUST be recomputed each step (online net weights
drift faster than transitions recycle through replay; storing
argmax at push time would feed stale-action data into the
projection).
* `forward_target(&self.sampled_h_tp1_d)` (was `&self.h_tp1_d`).
* `select_action_atoms(..., &self.sampled_next_actions_d, ...)`
(was `&self.next_actions_d`).
* `project_bellman_target(..., &self.sampled_rewards_d,
&self.sampled_dones_d, ...)` (was `rewards_d` / `dones_d`).
* `backward_logits(..., &self.sampled_actions_d, ...,
&mut self.td_per_sample_d, ...)` (added per-sample loss output).
* `backward_to_w_b_h(&self.sampled_h_t_d, ...)` (was `h_t_borrow`).
* Q grad_h_t accumulation REMOVED from Step 10 (stop-grad).
## Test: r7d_per_wiring.rs (gate G6)
Three invariants per `pearl_tests_must_prove_not_lock_observations`:
1. `replay.len()` grows by exactly `b_size` per `step_with_lobsim`
call (push semantics).
2. `sample_indices(b_size, α)` returns vec of length `b_size` on a
non-empty buffer.
3. Buffer caps at `per_capacity` (ring-with-random-replacement).
Drives 15 steps with `per_capacity=8`, asserts growth 0→5→8 across
the cap boundary.
## Acceptable host traffic this commit adds
* Per-step DtoH of 4 × b_size scalars (action/reward/done/log_pi_old)
for PER push metadata.
* Per-step DtoH of b_size floats (td_per_sample_d) for
update_priorities.
* Per-step HtoD of 3 × b_size scalars (sampled action/reward/done)
for sampled metadata gather.
* Per-step DtoD of 2 × b_size × HIDDEN_DIM floats (per-batch h_t /
h_tp1 slices) for PER push + gather.
PER bookkeeping is a control-plane operation by design (host-side
priority/index management); the device-side training hot path
(encoder, Q/π/V forward/backward, Adam) stays GPU-pure. GPU sum-tree
+ device-resident transitions are a Phase R-future optimization
flagged in `replay.rs`'s doc.
## What's NOT in this commit
* Q `loss_per_batch [B]` is now wired through `backward_logits` but
the DtoH happens inside step_with_lobsim (not inside
step_synthetic). Earlier R7d sketches considered a separate
`dqn_offpolicy_step` method; the in-step_synthetic redirect
approach landed because it touches fewer lines + reuses the
existing scratch buffer allocations + matches the trainer's
established λ-weighted multi-head pattern. A future refactor
could split for clarity.
Local sm_86 smoke gates: `cargo test -p ml-alpha --test
r7d_per_wiring -- --ignored --nocapture` (G6) +
`integrated_trainer_smoke` (end-to-end). Cluster smoke deferred to
R9.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Honors R6's commit-message promise to close the host-work boundary
the partial GPU-purity left behind. After R7a, step_with_lobsim's
post-fill pipeline is GPU-resident through the entire training step
except for 3 small host-slice uploads (actions, next_actions,
log_pi_old) that R7b lifts via R4's Thompson/argmax/log_pi kernels.
CHANGES:
1. New cuda/abs_copy.cu — element-wise dst[b] = fabsf(src[b]).
Feeds the |reward| signal into ema_update_on_done for the
MEAN_ABS_PNL_EMA slot without mutating signed rewards_d.
2. New trainer-owned per-step device buffers (allocated once in
new(), reused every step — no per-call churn):
reward_abs_d, actions_d, next_actions_d, log_pi_old_d,
advantages_d, returns_d
3. step_synthetic signature change: drops the 7 synthetic_* host
slice args (synthetic_actions, synthetic_rewards, synthetic_dones,
synthetic_next_actions, synthetic_advantages, synthetic_returns,
synthetic_log_pi_old). The body now reads from trainer-owned
device buffers (self.actions_d, self.rewards_d, etc.) via the
disjoint-field borrow rule. The 7 upload_i32/upload_f32 calls
are deleted — caller (step_with_lobsim) populates the buffers
via GPU kernels before invoking step_synthetic.
4. step_with_lobsim Step 6 rewritten end-to-end:
- Upload host Thompson outputs (actions, next_actions, log_pi_old)
to trainer buffers — R7b removes these via R4's GPU kernels.
- GPU abs_copy(rewards_d) → reward_abs_d.
- GPU ema_update_on_done(MEAN_ABS_PNL_EMA, reward_abs_d, dones_d).
- GPU launch_rl_controllers_per_step (R5) — all 7 controllers
adapt to this step's EMA inputs.
- GPU apply_reward_scale(rewards_d) in-place — reads the freshly
updated ISV[406] from the controller.
- GPU compute_advantage_return(rewards_d, dones_d, v_t, v_tp1)
→ returns_d, advantages_d.
- step_synthetic(snapshots) consumes all trainer device buffers,
runs the training kernel chain, returns stats.
- dqn_head.soft_update_target(isv_d) — R5 target-net Polyak
update with τ from ISV[401].
R7a PARTIAL — work R7b lifts:
- Host Thompson sampler still produces actions/next_actions/log_pi
(R7b replaces with R4 rl_action_kernel + argmax_expected_q +
log_pi_at_action — eliminating the 3 remaining HtoD uploads).
- v_pred_host → v_pred_d HtoD round-trip (the host already has
v_pred_host from the action-sampling Thompson read; R7b keeps V
on device throughout).
- v_tp1_d uses v_pred_host (V(s_t) approximated as V(s_{t+1}) until
R7b wires next_snapshots + forward_encoder(next_snapshots)).
The 4 final DtoH copies the R6 commit message warned about are
GONE. Hot path: snapshot upload (boundary HtoD), ISV diagnostic
readback (HEALTH_DIAG), and 3 host-Thompson uploads (R7b removes).
cargo check + cargo build --tests on ml-alpha green. Tests still
compile against the new step_synthetic signature (no test calls it
directly — only step_with_lobsim does).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
