The confidence gate and FRD gate were captured as no-ops during graph
warmup (step 1, before warmup threshold). Graph replay then never
fired them. Moving outside the capture region makes them execute as
individual kernel launches every step.
Result: Hold=100% from step 200-1000 (gate enforcing surfer Hold
default). Gate fires on nearly every non-Hold action until Q
confidence exceeds threshold=0.3.
Hold still drops after ~1500 steps as Q becomes confident — need
higher/adaptive threshold for sustained Hold protection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Confidence gate now gates ALL non-Hold actions (was opening-only)
- Gate warmup reduced 10000 → 100 steps, threshold raised 0.01 → 0.3
- log_pi_at_action moved to AFTER gates in all paths so PPO trains
on the post-gate action (gated-to-Hold gets log π(Hold))
- Hold prior advantage (ISV-driven, 0.1) for non-done steps
- Entropy gradient in PPO backward (fires all batch elements)
- Entropy controller: COEF_MAX=0.5, COEF_FLOOR=0.01, emergency bypass
Hold still collapses to 0% — investigating gate effectiveness.
gated_count shows low fire rate despite threshold=0.3 and near-uniform
Q distributions. The conf computation or gated action writing may
have a bug.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Hold prior (ISV-driven, per-step +adv for Hold, -adv for non-Hold)
doesn't prevent collapse. PPO's importance ratio mechanism amplifies
done-step gradients exponentially, overwhelming linear hold prior.
Entropy controller upgraded: COEF_MAX=0.5, symmetric response with
COEF_FLOOR=0.01, emergency bypass when entropy < 50% target.
Entropy gradient added to PPO backward. Still insufficient.
Key finding: confidence_gate and frd_gate are configured but not
firing (gated_count=0). These gates should enforce surfer philosophy
at action-selection level. Investigating next.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Added entropy gradient (∂H/∂logits) to PPO backward, fires on ALL
batch elements. Fixed controller: COEF_FLOOR=0.01, COEF_MAX=0.5,
emergency bypass Wiener when entropy < 50% target. Symmetric response
(non-zero coef even when entropy above target).
Still collapses: PPO done-gated gradient concentrates on trading
actions, overwhelming the diffuse entropy gradient. Hold drops to 0%
despite coef=0.5. The surfer philosophy needs structural enforcement
at the action-selection level, not just gradient-level entropy bonus.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Gate both PPO advantage and Q→π distillation on done-steps only:
- advantage = done * (returns - V) — non-done advantages are zero
- distillation gradient only fires when done > 0.5 per batch element
With 5-6% of batch having real rewards, non-done gradient was 94%
noise that destabilized Q-π alignment. Done-gating ensures π only
learns from steps where trades closed and rewards validate the signal.
qpa: holds at +0.625 through 3000 steps (previously crashed to -0.85).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Intermediate state — masked advantages still crash qpa because
distillation + entropy gradients run every step on non-done elements.
Next: two-timescale π update (option 4).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
With 5-6% of batch elements having real rewards per step, advantage
normalization (zero mean, unit var) turned 94% of the gradient into
noise. The /B gradient normalization already handles batch-size
invariance. Raw V-advantage bounded by reward_scale is sufficient.
Removed: normalize_advantages kernel (#if 0 in .cu), field, load,
both launch sites, q_logits_target_st_d (Q-advantage dead code).
ISV-driven TAU_MAX (slot 573, bootstrap 0.005) retained for future use.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Q-advantage (E_Q - V) caused phase-lag oscillation: π chased Q's
rapidly-shifting rankings even with target-Q and τ=0.005. The
feedback loop (π moves → rewards change → Q changes → advantages
flip) made qpa crash to -0.8 within a few thousand steps regardless
of target-net stability.
V-advantage (returns - V) with advantage normalization (zero mean,
unit var) is stable: l_pi=60 (not exploding), wr=0.32 (profitable),
training dynamics don't degrade over time.
qpa is negative with V-advantage — this is expected since π optimizes
PPO (reward-based) while Q optimizes C51 Bellman (distributional).
They have structurally different objectives; disagreement is normal.
Also: TAU_MAX is now ISV-driven (slot 573, bootstrap 0.005) in both
standalone and fused controller kernels. Removes hardcoded 0.05f.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PPO forward loss and DQN CE loss atomicAdd now divide by B before
accumulation. Loss values in JSONL are now means, not sums — same
magnitude regardless of batch size.
l_q: thousands → 26, l_pi: thousands → 18 at b=16.
Same values expected at b=256 and b=1024.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All RL head gradients flowing into the shared encoder (π, V, FRD,
outcome) are now scaled by 1/B at the grad_h_accumulate point.
Without this, doubling batch size doubles the encoder gradient from
each head, creating batch-size-dependent training dynamics.
Per-head Adam optimizers are self-normalizing (m/sqrt(v)), so their
weight gradients don't need explicit /B. Only the shared encoder
accumulation point matters.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Divide PPO surrogate gradient and Q→π distillation gradient by B in
their respective CUDA kernels. Without /B, doubling batch size doubles
gradient magnitude (implicit LR doubling), explaining why b=512
degraded vs b=256.
l_pi: 1170 → 684. Same LR now works across any batch size.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add normalize_advantages kernel: zero mean, unit variance across batch
via block tree-reduce. Launched after compute_advantage_return at both
call sites (reward pipeline + method).
Without normalization, Q-advantage (E_Q - V) produced all-positive or
all-negative advantages that made PPO reinforce/suppress all actions
uniformly. l_pi exploded to 454k, qpa crashed to -0.95.
With normalization: l_pi=1170 (stable), qpa rises from -0.005 to +0.584
in 500 steps. PPO now discriminates between actions based on relative
Q-value, not absolute magnitude.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace V-advantage (returns - V) with Q-advantage (E_Q(s,a) - V(s))
in compute_advantage_return kernel. Q's C51 distributional expected
value for the taken action feeds directly into PPO's advantage signal.
Root cause: with V-advantage, negative rewards pushed π AWAY from
Q-Thompson's preferred actions (q_pi_agree → -0.84). With Q-advantage,
the advantage is positive when Q rates the taken action above V's
baseline, aligning PPO and Q objectives.
Result: q_pi_agree goes from -0.84 (anti-correlated) to +0.31
(aligned) in 500 steps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Raw aux loss sum was 11M (unnormalized across B×K positions). Now
divides by n_valid per direction (prof/size), matching the original
step_batched normalization. aux: 11M → 116.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
BCE and aux losses were hardcoded to 0 in the GPU data path. Now
step_batched_from_device returns (bce_loss, aux_loss), stored as
last_bce_loss / last_aux_loss and fed into step stats + JSONL.
Event-based sync replaces host-blocking stream sync:
- perception: raw_event_record after training graph, raw_event_sync
in read_deferred_stats (instant — graph completed long ago)
- diag_staging: raw_event_record after snapshot_async, raw_event_sync
in sync_and_swap (instant — copies completed during GPU work)
All JSONL fields preserved. Loss values one-step deferred (same as
before for Q/π/V, now also for BCE/aux).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add apply_snapshot_from_device to RlLobBackend: DtoD copy from
perception SoA buffers into lobsim book arrays + book_update kernel.
No host roundtrip, no sync (stream-ordered).
step_with_lobsim_gpu now calls apply_snapshot_from_device with batch 0's
last-snapshot (K-1) book data. Produces non-zero dones, pnl, wr.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert + upload one file at a time instead of all 45M snapshots at
once. Peak host memory drops from ~10GB (all converted snapshots) to
~2GB (one file). Fixes OOMKilled on 40Gi L40S pods.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upload all pre-converted snapshots + FRD labels to GPU at init (1.2GB
for 5.6M snapshots / 2 files locally, ~11GB for 45M / 9 files on L40S).
New GPU kernels: gpu_sample_and_gather (PRNG sampling + AoS-to-SoA),
gpu_gather_next/gpu_gather_current (anchor offset re-gather),
gpu_gather_frd_labels (per-horizon label gather).
step_with_lobsim_gpu: GPU-only data path replacing host-side loader.
Encoder forwards via forward_encoder_from_device. Eliminates 7700 heap
allocs + 418k scalar copies + 16ms CPU work per step at b=256.
Init uploads use cuMemcpyHtoD_v2 (synchronous, one-time).
Note: apply_snapshot skipped (lobsim book stale, dones/rewards=0).
Follow-up: copy last-snapshot book data from SoA to lobsim buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
diag_staging.sync_and_swap() was at the START of the step loop,
blocking host for ~5.5ms while GPU was idle. Moved to AFTER
step_with_lobsim (which ends with end-of-step sync that guarantees
previous diag copies completed). Eliminates one of the two per-step
host-blocking syncs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Background thread runs loader.next_sequence_pair() × n_batch while
GPU trains on the current batch. sync_channel(2) double-buffer.
Eliminates loader latency from the critical path at b=256 where
256 sequential next_sequence_pair calls took ~19ms/step (the entire
step budget at 52 sps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New snapshot_aos_to_soa.cu kernel: one thread per snapshot, reads
contiguous Mbp10RawInput structs from mapped-pinned AoS buffer,
scatters fields to 10 SoA device buffers. Replaces 418k scalar
host copies + 10 DtoD memcpys with single memcpy + one kernel launch.
Add #[repr(C)] to Mbp10RawInput with 216-byte compile-time assertion.
Single MappedRecordBuffer replaces 10 separate staging buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split forward_only into forward_only_dispatch (no sync, no download)
and forward_only (sync + download). forward_encoder now calls the
dispatch-only path — it only needs h_t_d on device, stream-ordered.
Was: 2 × (raw_stream_sync + probs download) per step from the two
forward_encoder calls in step_with_lobsim. Each sync drained the
entire GPU pipeline while CPU did host staging for the next call.
GPU was 98.5% idle per nsys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
dqn_distributional_q_bwd: restructure from 21 threads/block (Q_N_ATOMS)
to 256 threads/block. Each thread maps to one (action, atom) pair.
Softmax reduction stays in warp 0; all 231 threads write gradients in
parallel. Block tree-reduce for loss accumulation (no intra-block
atomicAdd). Was 23% of L40S GPU time at 167μs/call.
Convert ALL remaining 46 launch_builder calls to raw_launch across
dqn.rs (9), ppo.rs (6), outcome_head.rs (7), frd.rs (4), noisy.rs (3),
bucket_routing.rs (5), cfc/step.rs, aux_trunk.rs, snap_features.rs,
loss.rs. Zero launch_builder in crate (1 comment only).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 83 device_ptr()/device_ptr_mut() calls with raw_ptr() across
perception.rs (65), integrated.rs (8), loss.rs (3), cfc/step.rs (2),
snap_features.rs (2), isv.rs (2), mamba2_block.rs (1). Each device_ptr
created a SyncOnDrop guard with cuEventRecord overhead.
Also converts remaining cudarc memcpy_dtod_async and .synchronize()
to raw equivalents. Fix unused mut/import warnings.
Note: nsys reveals the REAL bottleneck is CudaSlice::drop() calling
stream.synchronize() before cuMemFree (has_async_alloc=false). 956
temp alloc drops over 200 steps = ~5 syncs/step from drops alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert 110 remaining launch_builder calls across perception.rs (74),
mamba2_block.rs (31), optim.rs (2), heads.rs (3) to raw_launch() with
RawArgs. Also convert device_ptr()/memcpy_dtod_async/synchronize to
raw equivalents. Zero cudarc driver API in the entire step pipeline.
Combined with the previous 61 integrated.rs conversions, every kernel
launch in the training loop now bypasses cudarc's event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert every kernel launch to raw_launch() with RawArgs — zero cudarc
driver API in the steady-state step loop. Cache raw_stream/raw_train_stream
CUstream handles at init. Replace all stream.cu_stream() with cached
field reads. This eliminates ~122 spurious cuStreamWaitEvent+cuEventRecord
calls per step from cudarc's multi-stream event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Force cudarc's is_managing_stream_synchronization() to false.
Eliminates ALL internal event tracking (cuStreamSynchronize,
cuEventRecord, cuStreamWaitEvent) that was adding 18.7 syncs/step.
All stream synchronization is managed manually via raw_launch events.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace fatbin with single-arch cubin auto-detected via nvidia-smi.
Each node compiles for its own GPU — faster builds, guaranteed compat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd).
Delete: isv_host Vec, isv_staging, loss_staging, loss_dtod_done_event,
all DtoD copies for ISV/losses, all event record/sync for losses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd) and 2 entropy accumulators.
Delete: isv_host Vec, isv_staging, frd_loss_staging, scalar_staging,
loss_staging_3, loss_dtod_done_event, all DtoD copies for ISV/losses,
all event record/sync for losses.
ISV is ~2.3KB, losses are 24B — PCIe bandwidth penalty negligible.
Mapped-pinned dev_ptr is stable → CUDA graph capture unaffected.
External head methods (DqnHead, PolicyHead, ValueHead, FrdHead)
changed from &CudaSlice<f32> to &u64 for ISV/loss device pointers —
.arg(&u64) pushes the same 8-byte CUdeviceptr as .arg(&CudaSlice).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace all cudarc graph.launch(), event.record/synchronize(),
stream.wait(), and memset_zeros in the hot path with direct CUDA
driver API calls. Eliminates cudarc's internal bind_to_thread +
event tracking overhead (~200us per call x ~40 calls/step).
Steady-state hot path is now zero cudarc calls:
- 4x graph.launch() → raw_graph_launch (cuGraphLaunch)
- 1x event.synchronize() → raw_event_sync (cuEventSynchronize)
- 5x event.record() → raw_event_record (cuEventRecord)
- 4x stream.wait() → raw_stream_wait_event (cuStreamWaitEvent)
- 28x memset_zeros → raw_memset_d8_zero (cuMemsetD8Async)
Also removed 36 redundant memset_zeros inside training graph
capture — backward kernels overwrite every element, so these
burned memset bandwidth on every graph replay for zero benefit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Async loss readback via loss_dtod_done_event — zero syncs in steady-
state step. Return previous step's cached stats (one-step delay).
DtoDs complete by stream ordering before next step's event wait.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 15 CudaSlice device_ptr() calls (each triggers cudarc event
tracking) with direct raw_ptr() u64 reads. Zero CUDA driver overhead
in the diag staging path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace -cubin -arch=sm_XX with -fatbin -gencode for all 3 GPU archs.
All include_bytes! paths updated from .cubin to .fatbin. Removes
CUDA_COMPUTE_CAP env var dependency. cuModuleLoadData transparently
selects the right arch from the fat binary at runtime.
Fixes CUDA_ERROR_NO_BINARY_FOR_GPU on H100 (sm_90).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wire outcome CE backward → grad_W/b → Adam → grad_h_accumulate.
Add PopArt, spectral, Q-bias, per-branch LR, outcome metrics to
diag JSONL for training validation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
One-command production run: push, submit, monitor, download, analyze.
Memory: cudarc overhead pearl, nsys workflow reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cache all hot-path device pointers as raw u64 in HotPathPtrs struct at
trainer init. Replaces 9 device_ptr()/device_ptr_mut() calls (each
triggers cudarc event wait+record → cuStreamSynchronize) with direct
u64 reads from the cached struct.
Affected call sites:
- ISV staging DtoD (step_synthetic)
- Batched loss readback: pi/Q/V/FRD DtoD copies (step_synthetic)
- h_t→h_tp1 encoder DtoD copy (step_with_lobsim)
- trade_context/multires_output ptr pass to perception encoder
All pointers are stable (buffers pre-allocated at init, never
reallocated). Utility functions (read_slice_d, write_slice_f32_d, etc.)
retain device_ptr() as they operate on arbitrary caller-provided buffers
and each already does its own stream sync.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All unconditional. No feature flags. ISV controls magnitudes.
PopArt normalize replaces apply_reward_scale in step_with_lobsim,
with V-correct on v_pred_d and v_pred_tp1_d. Spectral norm runs
one power iteration per step on DQN/IQN/policy weight matrices
after each Adam step in dqn_replay_step. Spectral decouple adds
lambda*mean(logits^2) penalty to Q and pi loss in step_synthetic.
K=3 outcome classifier forward+CE loss runs each step; labels
assigned from reward/done signals. Q-bias correction tracks
mean(Q - return) and emits correction to ISV. Per-branch LR
controller adjusts per-head LR scale factors from loss deltas.
ISV slots 553-572 bootstrapped. Per-branch LR kernel ABI fixed:
current losses passed as args (eliminates slot 572-575 collision
with RL_OUTCOME_AUX_LAMBDA_INDEX).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
