Static: β×log(π_ref(a)) added to reward, stored in PER. Q sees
consistent Hold preference across replays.
Dynamic: -β×log(π_θ(a)) added to advantage only (not stored in PER).
Provides entropy-regularized signal (SAC-like) that adapts to
current policy. Low π_θ → positive advantage (explore). High π_θ →
near-zero (don't over-concentrate).
Result: entropy STABLE 0.94-1.29 through 5000 steps (no collapse).
Hold 62-100%. wr=0.36 (best local). Removed KL gradient kernel
from training step (KL is now entirely in the reward/advantage).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the separate KL gradient kernel with an RLHF-style KL reward:
r_modified = r_env - β × (log π_θ(a|s) - log π_ref(a))
This feeds through PPO's standard advantage pipeline — no structural
imbalance between KL (100% of steps) and PPO (6% done-steps). The KL
signal has the same per-step coverage as the advantage because it IS
the advantage on non-done steps.
Remove done-gating: all steps now get advantages (KL reward provides
directional signal on non-done steps). Remove distillation done-gating
too (distill on all steps now that PPO has signal everywhere).
Result: entropy STABLE 0.81-1.31 (was collapsing to 0.0 with gradient
KL). Hold oscillates 62-100% at b=16 (expected — sparse PnL). At
b=1024 should settle ~50% with 60× denser PnL signal.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 24 cudarc .arg() calls (8+16) with RawArgs + raw_launch.
Eliminates ~48 cuStreamWaitEvent + cuEventRecord per step from
cudarc's event tracking. Last hot-path launch_builder in the GPU
RL training pipeline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
β=0.003 pinned Hold=100% at b=1024 despite 60 dones/step.
The KL gradient overwhelms even the dense PPO signal.
Try 6× lower β to find the balance point.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove unnecessary stream.synchronize() from:
1. step_fill_from_market_targets (lobsim): stream ordering handles
the dependency — step_pnl_track launches on the same stream
2. step_pnl_track (lobsim): downstream kernels read pos_d via
stream ordering, no host read needed per step
Defer pos_fraction readback by one step (same pattern as loss
readback) — eliminates the third 7ms sync. pos_fraction is a
dataset-level statistic that barely changes step to step.
nsys: 3.0 → 1.0 slow syncs/step. Sync time: 21.8 → 9.65 ms/step.
The remaining sync is the training graph event wait (actual compute).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Old RL_KL_TARGET_INDEX at slot 454 was for the Q-distill KL target.
New reference-policy KL target at slot 580 needs a distinct name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Controller observes KL(π_θ||π_ref) directly — zero feedback lag unlike
the Hold%-based controller that had 50-step delay. Schulman bounded
step at ±0.5%/step (much gentler than previous ±5%).
At b=16 (1 done/step) the PPO signal is too sparse to counterbalance
even tiny β — Hold pins at 100%. This is a small-batch artifact.
At b=1024 (60 dones/step) the 60× denser PPO signal should balance
with the KL controller for stable Hold~50%.
ISV slots: KL_TARGET=580 (bootstrap 0.5), REWARD_KL_BETA=579 (0.005).
Gradient β=578 (bootstrap 0.003), range [0.001, 0.1].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reward augmentation now ISV-driven (slot 579, bootstrap 0.005).
Gradient KL β reduced to 0.003 (gentle nudge, not straitjacket).
Adaptive β controller disabled — overshoots consistently.
At b=16 (1 done/step), KL dominates sparse PPO signal → Hold=100%.
At b=1024 (60 dones/step), the done-gated PPO signal is 60× denser
and should balance with the gentle KL without collapse. The b=1024
run without KL (done-gated only) showed qpa=0.84, ent=2.04, wr=0.34
stable at 25k steps — the best metrics yet.
Removed dead code from normalize_advantages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add KL bonus to reward: r += 0.02 × log(π_ref(a_taken)). This makes
Q learn the same KL-regularized objective as π, aligning their action
rankings (qpa positive when both value Hold). Reward written back to
rewards_d so PER replay also sees the augmented signal.
Decoupled from the adaptive gradient β: reward uses fixed 0.02,
gradient uses adaptive β from the hold_frac controller. Prevents
compounding that caused Hold=100% when both used the same β.
Removed dead code: normalize_advantages kernel (#if 0 block),
RL_HOLD_PRIOR_INDEX references.
Results at reward_kl_β=0.02: Hold 25-88% (oscillating around target),
entropy 1.19-1.45 (healthy), wr=0.35, qpa oscillates ±0.4.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The KL β controller adjusts β based on hold_frac_ema vs target:
- Hold drops below 40% → β ramps UP 5%/step (emergency)
- Hold above 60% → β decays DOWN 0.1%/step (slow relax)
- Asymmetric rates prevent oscillation
Result: Hold stabilizes at 31-62% (target 50%), entropy 1.3-1.5
(healthy), wr improves 0.15→0.33 over 5000 steps. The surfer waits
for the wave — ~50% of steps are Hold, trades are selective.
Bootstrap β=0.02, controller range [0.001, 1.0].
Hold_frac_ema α=0.05 for responsive tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add rl_kl_reference_grad kernel: β × (π_θ(a) - π_ref(a)) gradient
fires on ALL batch elements, ALL steps. π_ref = [Hold=50%, rest=5%]
encodes surfer philosophy as a continuous prior.
Unlike entropy (pushes to uniform, doesn't know Hold is special),
gates (binary, blocks learning), or hold-prior advantages (overwhelmed
by PPO ratio), KL penalty is smooth, continuous, and specifically
preserves the Hold-heavy reference distribution.
β=0.5 (ISV slot 578). Hold=75-100% through 5000 steps.
wr=0.339 — profitable trades while maintaining Hold dominance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The confidence gate and FRD gate were captured as no-ops during graph
warmup (step 1, before warmup threshold). Graph replay then never
fired them. Moving outside the capture region makes them execute as
individual kernel launches every step.
Result: Hold=100% from step 200-1000 (gate enforcing surfer Hold
default). Gate fires on nearly every non-Hold action until Q
confidence exceeds threshold=0.3.
Hold still drops after ~1500 steps as Q becomes confident — need
higher/adaptive threshold for sustained Hold protection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Confidence gate now gates ALL non-Hold actions (was opening-only)
- Gate warmup reduced 10000 → 100 steps, threshold raised 0.01 → 0.3
- log_pi_at_action moved to AFTER gates in all paths so PPO trains
on the post-gate action (gated-to-Hold gets log π(Hold))
- Hold prior advantage (ISV-driven, 0.1) for non-done steps
- Entropy gradient in PPO backward (fires all batch elements)
- Entropy controller: COEF_MAX=0.5, COEF_FLOOR=0.01, emergency bypass
Hold still collapses to 0% — investigating gate effectiveness.
gated_count shows low fire rate despite threshold=0.3 and near-uniform
Q distributions. The conf computation or gated action writing may
have a bug.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Hold prior (ISV-driven, per-step +adv for Hold, -adv for non-Hold)
doesn't prevent collapse. PPO's importance ratio mechanism amplifies
done-step gradients exponentially, overwhelming linear hold prior.
Entropy controller upgraded: COEF_MAX=0.5, symmetric response with
COEF_FLOOR=0.01, emergency bypass when entropy < 50% target.
Entropy gradient added to PPO backward. Still insufficient.
Key finding: confidence_gate and frd_gate are configured but not
firing (gated_count=0). These gates should enforce surfer philosophy
at action-selection level. Investigating next.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Added entropy gradient (∂H/∂logits) to PPO backward, fires on ALL
batch elements. Fixed controller: COEF_FLOOR=0.01, COEF_MAX=0.5,
emergency bypass Wiener when entropy < 50% target. Symmetric response
(non-zero coef even when entropy above target).
Still collapses: PPO done-gated gradient concentrates on trading
actions, overwhelming the diffuse entropy gradient. Hold drops to 0%
despite coef=0.5. The surfer philosophy needs structural enforcement
at the action-selection level, not just gradient-level entropy bonus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Gate both PPO advantage and Q→π distillation on done-steps only:
- advantage = done * (returns - V) — non-done advantages are zero
- distillation gradient only fires when done > 0.5 per batch element
With 5-6% of batch having real rewards, non-done gradient was 94%
noise that destabilized Q-π alignment. Done-gating ensures π only
learns from steps where trades closed and rewards validate the signal.
qpa: holds at +0.625 through 3000 steps (previously crashed to -0.85).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Intermediate state — masked advantages still crash qpa because
distillation + entropy gradients run every step on non-done elements.
Next: two-timescale π update (option 4).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With 5-6% of batch elements having real rewards per step, advantage
normalization (zero mean, unit var) turned 94% of the gradient into
noise. The /B gradient normalization already handles batch-size
invariance. Raw V-advantage bounded by reward_scale is sufficient.
Removed: normalize_advantages kernel (#if 0 in .cu), field, load,
both launch sites, q_logits_target_st_d (Q-advantage dead code).
ISV-driven TAU_MAX (slot 573, bootstrap 0.005) retained for future use.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Q-advantage (E_Q - V) caused phase-lag oscillation: π chased Q's
rapidly-shifting rankings even with target-Q and τ=0.005. The
feedback loop (π moves → rewards change → Q changes → advantages
flip) made qpa crash to -0.8 within a few thousand steps regardless
of target-net stability.
V-advantage (returns - V) with advantage normalization (zero mean,
unit var) is stable: l_pi=60 (not exploding), wr=0.32 (profitable),
training dynamics don't degrade over time.
qpa is negative with V-advantage — this is expected since π optimizes
PPO (reward-based) while Q optimizes C51 Bellman (distributional).
They have structurally different objectives; disagreement is normal.
Also: TAU_MAX is now ISV-driven (slot 573, bootstrap 0.005) in both
standalone and fused controller kernels. Removes hardcoded 0.05f.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PPO forward loss and DQN CE loss atomicAdd now divide by B before
accumulation. Loss values in JSONL are now means, not sums — same
magnitude regardless of batch size.
l_q: thousands → 26, l_pi: thousands → 18 at b=16.
Same values expected at b=256 and b=1024.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All RL head gradients flowing into the shared encoder (π, V, FRD,
outcome) are now scaled by 1/B at the grad_h_accumulate point.
Without this, doubling batch size doubles the encoder gradient from
each head, creating batch-size-dependent training dynamics.
Per-head Adam optimizers are self-normalizing (m/sqrt(v)), so their
weight gradients don't need explicit /B. Only the shared encoder
accumulation point matters.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Divide PPO surrogate gradient and Q→π distillation gradient by B in
their respective CUDA kernels. Without /B, doubling batch size doubles
gradient magnitude (implicit LR doubling), explaining why b=512
degraded vs b=256.
l_pi: 1170 → 684. Same LR now works across any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add normalize_advantages kernel: zero mean, unit variance across batch
via block tree-reduce. Launched after compute_advantage_return at both
call sites (reward pipeline + method).
Without normalization, Q-advantage (E_Q - V) produced all-positive or
all-negative advantages that made PPO reinforce/suppress all actions
uniformly. l_pi exploded to 454k, qpa crashed to -0.95.
With normalization: l_pi=1170 (stable), qpa rises from -0.005 to +0.584
in 500 steps. PPO now discriminates between actions based on relative
Q-value, not absolute magnitude.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace V-advantage (returns - V) with Q-advantage (E_Q(s,a) - V(s))
in compute_advantage_return kernel. Q's C51 distributional expected
value for the taken action feeds directly into PPO's advantage signal.
Root cause: with V-advantage, negative rewards pushed π AWAY from
Q-Thompson's preferred actions (q_pi_agree → -0.84). With Q-advantage,
the advantage is positive when Q rates the taken action above V's
baseline, aligning PPO and Q objectives.
Result: q_pi_agree goes from -0.84 (anti-correlated) to +0.31
(aligned) in 500 steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Raw aux loss sum was 11M (unnormalized across B×K positions). Now
divides by n_valid per direction (prof/size), matching the original
step_batched normalization. aux: 11M → 116.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
BCE and aux losses were hardcoded to 0 in the GPU data path. Now
step_batched_from_device returns (bce_loss, aux_loss), stored as
last_bce_loss / last_aux_loss and fed into step stats + JSONL.
Event-based sync replaces host-blocking stream sync:
- perception: raw_event_record after training graph, raw_event_sync
in read_deferred_stats (instant — graph completed long ago)
- diag_staging: raw_event_record after snapshot_async, raw_event_sync
in sync_and_swap (instant — copies completed during GPU work)
All JSONL fields preserved. Loss values one-step deferred (same as
before for Q/π/V, now also for BCE/aux).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add apply_snapshot_from_device to RlLobBackend: DtoD copy from
perception SoA buffers into lobsim book arrays + book_update kernel.
No host roundtrip, no sync (stream-ordered).
step_with_lobsim_gpu now calls apply_snapshot_from_device with batch 0's
last-snapshot (K-1) book data. Produces non-zero dones, pnl, wr.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert + upload one file at a time instead of all 45M snapshots at
once. Peak host memory drops from ~10GB (all converted snapshots) to
~2GB (one file). Fixes OOMKilled on 40Gi L40S pods.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upload all pre-converted snapshots + FRD labels to GPU at init (1.2GB
for 5.6M snapshots / 2 files locally, ~11GB for 45M / 9 files on L40S).
New GPU kernels: gpu_sample_and_gather (PRNG sampling + AoS-to-SoA),
gpu_gather_next/gpu_gather_current (anchor offset re-gather),
gpu_gather_frd_labels (per-horizon label gather).
step_with_lobsim_gpu: GPU-only data path replacing host-side loader.
Encoder forwards via forward_encoder_from_device. Eliminates 7700 heap
allocs + 418k scalar copies + 16ms CPU work per step at b=256.
Init uploads use cuMemcpyHtoD_v2 (synchronous, one-time).
Note: apply_snapshot skipped (lobsim book stale, dones/rewards=0).
Follow-up: copy last-snapshot book data from SoA to lobsim buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
diag_staging.sync_and_swap() was at the START of the step loop,
blocking host for ~5.5ms while GPU was idle. Moved to AFTER
step_with_lobsim (which ends with end-of-step sync that guarantees
previous diag copies completed). Eliminates one of the two per-step
host-blocking syncs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Background thread runs loader.next_sequence_pair() × n_batch while
GPU trains on the current batch. sync_channel(2) double-buffer.
Eliminates loader latency from the critical path at b=256 where
256 sequential next_sequence_pair calls took ~19ms/step (the entire
step budget at 52 sps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New snapshot_aos_to_soa.cu kernel: one thread per snapshot, reads
contiguous Mbp10RawInput structs from mapped-pinned AoS buffer,
scatters fields to 10 SoA device buffers. Replaces 418k scalar
host copies + 10 DtoD memcpys with single memcpy + one kernel launch.
Add #[repr(C)] to Mbp10RawInput with 216-byte compile-time assertion.
Single MappedRecordBuffer replaces 10 separate staging buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split forward_only into forward_only_dispatch (no sync, no download)
and forward_only (sync + download). forward_encoder now calls the
dispatch-only path — it only needs h_t_d on device, stream-ordered.
Was: 2 × (raw_stream_sync + probs download) per step from the two
forward_encoder calls in step_with_lobsim. Each sync drained the
entire GPU pipeline while CPU did host staging for the next call.
GPU was 98.5% idle per nsys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
dqn_distributional_q_bwd: restructure from 21 threads/block (Q_N_ATOMS)
to 256 threads/block. Each thread maps to one (action, atom) pair.
Softmax reduction stays in warp 0; all 231 threads write gradients in
parallel. Block tree-reduce for loss accumulation (no intra-block
atomicAdd). Was 23% of L40S GPU time at 167μs/call.
Convert ALL remaining 46 launch_builder calls to raw_launch across
dqn.rs (9), ppo.rs (6), outcome_head.rs (7), frd.rs (4), noisy.rs (3),
bucket_routing.rs (5), cfc/step.rs, aux_trunk.rs, snap_features.rs,
loss.rs. Zero launch_builder in crate (1 comment only).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 83 device_ptr()/device_ptr_mut() calls with raw_ptr() across
perception.rs (65), integrated.rs (8), loss.rs (3), cfc/step.rs (2),
snap_features.rs (2), isv.rs (2), mamba2_block.rs (1). Each device_ptr
created a SyncOnDrop guard with cuEventRecord overhead.
Also converts remaining cudarc memcpy_dtod_async and .synchronize()
to raw equivalents. Fix unused mut/import warnings.
Note: nsys reveals the REAL bottleneck is CudaSlice::drop() calling
stream.synchronize() before cuMemFree (has_async_alloc=false). 956
temp alloc drops over 200 steps = ~5 syncs/step from drops alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert 110 remaining launch_builder calls across perception.rs (74),
mamba2_block.rs (31), optim.rs (2), heads.rs (3) to raw_launch() with
RawArgs. Also convert device_ptr()/memcpy_dtod_async/synchronize to
raw equivalents. Zero cudarc driver API in the entire step pipeline.
Combined with the previous 61 integrated.rs conversions, every kernel
launch in the training loop now bypasses cudarc's event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert every kernel launch to raw_launch() with RawArgs — zero cudarc
driver API in the steady-state step loop. Cache raw_stream/raw_train_stream
CUstream handles at init. Replace all stream.cu_stream() with cached
field reads. This eliminates ~122 spurious cuStreamWaitEvent+cuEventRecord
calls per step from cudarc's multi-stream event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Force cudarc's is_managing_stream_synchronization() to false.
Eliminates ALL internal event tracking (cuStreamSynchronize,
cuEventRecord, cuStreamWaitEvent) that was adding 18.7 syncs/step.
All stream synchronization is managed manually via raw_launch events.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace fatbin with single-arch cubin auto-detected via nvidia-smi.
Each node compiles for its own GPU — faster builds, guaranteed compat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd).
Delete: isv_host Vec, isv_staging, loss_staging, loss_dtod_done_event,
all DtoD copies for ISV/losses, all event record/sync for losses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd) and 2 entropy accumulators.
Delete: isv_host Vec, isv_staging, frd_loss_staging, scalar_staging,
loss_staging_3, loss_dtod_done_event, all DtoD copies for ISV/losses,
all event record/sync for losses.
ISV is ~2.3KB, losses are 24B — PCIe bandwidth penalty negligible.
Mapped-pinned dev_ptr is stable → CUDA graph capture unaffected.
External head methods (DqnHead, PolicyHead, ValueHead, FrdHead)
changed from &CudaSlice<f32> to &u64 for ISV/loss device pointers —
.arg(&u64) pushes the same 8-byte CUdeviceptr as .arg(&CudaSlice).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace all cudarc graph.launch(), event.record/synchronize(),
stream.wait(), and memset_zeros in the hot path with direct CUDA
driver API calls. Eliminates cudarc's internal bind_to_thread +
event tracking overhead (~200us per call x ~40 calls/step).
Steady-state hot path is now zero cudarc calls:
- 4x graph.launch() → raw_graph_launch (cuGraphLaunch)
- 1x event.synchronize() → raw_event_sync (cuEventSynchronize)
- 5x event.record() → raw_event_record (cuEventRecord)
- 4x stream.wait() → raw_stream_wait_event (cuStreamWaitEvent)
- 28x memset_zeros → raw_memset_d8_zero (cuMemsetD8Async)
Also removed 36 redundant memset_zeros inside training graph
capture — backward kernels overwrite every element, so these
burned memset bandwidth on every graph replay for zero benefit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Async loss readback via loss_dtod_done_event — zero syncs in steady-
state step. Return previous step's cached stats (one-step delay).
DtoDs complete by stream ordering before next step's event wait.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 15 CudaSlice device_ptr() calls (each triggers cudarc event
tracking) with direct raw_ptr() u64 reads. Zero CUDA driver overhead
in the diag staging path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
