Add apply_snapshot_from_device to RlLobBackend: DtoD copy from
perception SoA buffers into lobsim book arrays + book_update kernel.
No host roundtrip, no sync (stream-ordered).
step_with_lobsim_gpu now calls apply_snapshot_from_device with batch 0's
last-snapshot (K-1) book data. Produces non-zero dones, pnl, wr.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert + upload one file at a time instead of all 45M snapshots at
once. Peak host memory drops from ~10GB (all converted snapshots) to
~2GB (one file). Fixes OOMKilled on 40Gi L40S pods.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Upload all pre-converted snapshots + FRD labels to GPU at init (1.2GB
for 5.6M snapshots / 2 files locally, ~11GB for 45M / 9 files on L40S).
New GPU kernels: gpu_sample_and_gather (PRNG sampling + AoS-to-SoA),
gpu_gather_next/gpu_gather_current (anchor offset re-gather),
gpu_gather_frd_labels (per-horizon label gather).
step_with_lobsim_gpu: GPU-only data path replacing host-side loader.
Encoder forwards via forward_encoder_from_device. Eliminates 7700 heap
allocs + 418k scalar copies + 16ms CPU work per step at b=256.
Init uploads use cuMemcpyHtoD_v2 (synchronous, one-time).
Note: apply_snapshot skipped (lobsim book stale, dones/rewards=0).
Follow-up: copy last-snapshot book data from SoA to lobsim buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
diag_staging.sync_and_swap() was at the START of the step loop,
blocking host for ~5.5ms while GPU was idle. Moved to AFTER
step_with_lobsim (which ends with end-of-step sync that guarantees
previous diag copies completed). Eliminates one of the two per-step
host-blocking syncs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Background thread runs loader.next_sequence_pair() × n_batch while
GPU trains on the current batch. sync_channel(2) double-buffer.
Eliminates loader latency from the critical path at b=256 where
256 sequential next_sequence_pair calls took ~19ms/step (the entire
step budget at 52 sps).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New snapshot_aos_to_soa.cu kernel: one thread per snapshot, reads
contiguous Mbp10RawInput structs from mapped-pinned AoS buffer,
scatters fields to 10 SoA device buffers. Replaces 418k scalar
host copies + 10 DtoD memcpys with single memcpy + one kernel launch.
Add #[repr(C)] to Mbp10RawInput with 216-byte compile-time assertion.
Single MappedRecordBuffer replaces 10 separate staging buffers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split forward_only into forward_only_dispatch (no sync, no download)
and forward_only (sync + download). forward_encoder now calls the
dispatch-only path — it only needs h_t_d on device, stream-ordered.
Was: 2 × (raw_stream_sync + probs download) per step from the two
forward_encoder calls in step_with_lobsim. Each sync drained the
entire GPU pipeline while CPU did host staging for the next call.
GPU was 98.5% idle per nsys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
dqn_distributional_q_bwd: restructure from 21 threads/block (Q_N_ATOMS)
to 256 threads/block. Each thread maps to one (action, atom) pair.
Softmax reduction stays in warp 0; all 231 threads write gradients in
parallel. Block tree-reduce for loss accumulation (no intra-block
atomicAdd). Was 23% of L40S GPU time at 167μs/call.
Convert ALL remaining 46 launch_builder calls to raw_launch across
dqn.rs (9), ppo.rs (6), outcome_head.rs (7), frd.rs (4), noisy.rs (3),
bucket_routing.rs (5), cfc/step.rs, aux_trunk.rs, snap_features.rs,
loss.rs. Zero launch_builder in crate (1 comment only).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 83 device_ptr()/device_ptr_mut() calls with raw_ptr() across
perception.rs (65), integrated.rs (8), loss.rs (3), cfc/step.rs (2),
snap_features.rs (2), isv.rs (2), mamba2_block.rs (1). Each device_ptr
created a SyncOnDrop guard with cuEventRecord overhead.
Also converts remaining cudarc memcpy_dtod_async and .synchronize()
to raw equivalents. Fix unused mut/import warnings.
Note: nsys reveals the REAL bottleneck is CudaSlice::drop() calling
stream.synchronize() before cuMemFree (has_async_alloc=false). 956
temp alloc drops over 200 steps = ~5 syncs/step from drops alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert 110 remaining launch_builder calls across perception.rs (74),
mamba2_block.rs (31), optim.rs (2), heads.rs (3) to raw_launch() with
RawArgs. Also convert device_ptr()/memcpy_dtod_async/synchronize to
raw equivalents. Zero cudarc driver API in the entire step pipeline.
Combined with the previous 61 integrated.rs conversions, every kernel
launch in the training loop now bypasses cudarc's event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert every kernel launch to raw_launch() with RawArgs — zero cudarc
driver API in the steady-state step loop. Cache raw_stream/raw_train_stream
CUstream handles at init. Replace all stream.cu_stream() with cached
field reads. This eliminates ~122 spurious cuStreamWaitEvent+cuEventRecord
calls per step from cudarc's multi-stream event tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Force cudarc's is_managing_stream_synchronization() to false.
Eliminates ALL internal event tracking (cuStreamSynchronize,
cuEventRecord, cuStreamWaitEvent) that was adding 18.7 syncs/step.
All stream synchronization is managed manually via raw_launch events.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sets CUDA_COMPUTE_CAP dynamically from the GPU on the node. Clears
both ml-alpha and ml-core build caches. Captures build.rs stderr in
pod logs for debugging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace fatbin with single-arch cubin auto-detected via nvidia-smi.
Each node compiles for its own GPU — faster builds, guaranteed compat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd).
Delete: isv_host Vec, isv_staging, loss_staging, loss_dtod_done_event,
all DtoD copies for ISV/losses, all event record/sync for losses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace isv_d CudaSlice with isv_mapped MappedF32Buffer. GPU writes
via dev_ptr, host reads via host_ptr. Zero copies, zero syncs.
Same for 4 loss accumulators (q/pi/v/frd) and 2 entropy accumulators.
Delete: isv_host Vec, isv_staging, frd_loss_staging, scalar_staging,
loss_staging_3, loss_dtod_done_event, all DtoD copies for ISV/losses,
all event record/sync for losses.
ISV is ~2.3KB, losses are 24B — PCIe bandwidth penalty negligible.
Mapped-pinned dev_ptr is stable → CUDA graph capture unaffected.
External head methods (DqnHead, PolicyHead, ValueHead, FrdHead)
changed from &CudaSlice<f32> to &u64 for ISV/loss device pointers —
.arg(&u64) pushes the same 8-byte CUdeviceptr as .arg(&CudaSlice).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace all cudarc graph.launch(), event.record/synchronize(),
stream.wait(), and memset_zeros in the hot path with direct CUDA
driver API calls. Eliminates cudarc's internal bind_to_thread +
event tracking overhead (~200us per call x ~40 calls/step).
Steady-state hot path is now zero cudarc calls:
- 4x graph.launch() → raw_graph_launch (cuGraphLaunch)
- 1x event.synchronize() → raw_event_sync (cuEventSynchronize)
- 5x event.record() → raw_event_record (cuEventRecord)
- 4x stream.wait() → raw_stream_wait_event (cuStreamWaitEvent)
- 28x memset_zeros → raw_memset_d8_zero (cuMemsetD8Async)
Also removed 36 redundant memset_zeros inside training graph
capture — backward kernels overwrite every element, so these
burned memset bandwidth on every graph replay for zero benefit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Async loss readback via loss_dtod_done_event — zero syncs in steady-
state step. Return previous step's cached stats (one-step delay).
DtoDs complete by stream ordering before next step's event wait.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 15 CudaSlice device_ptr() calls (each triggers cudarc event
tracking) with direct raw_ptr() u64 reads. Zero CUDA driver overhead
in the diag staging path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace -cubin -arch=sm_XX with -fatbin -gencode for all 3 GPU archs.
All include_bytes! paths updated from .cubin to .fatbin. Removes
CUDA_COMPUTE_CAP env var dependency. cuModuleLoadData transparently
selects the right arch from the fat binary at runtime.
Fixes CUDA_ERROR_NO_BINARY_FOR_GPU on H100 (sm_90).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wire outcome CE backward → grad_W/b → Adam → grad_h_accumulate.
Add PopArt, spectral, Q-bias, per-branch LR, outcome metrics to
diag JSONL for training validation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
One-command production run: push, submit, monitor, download, analyze.
Memory: cudarc overhead pearl, nsys workflow reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cache all hot-path device pointers as raw u64 in HotPathPtrs struct at
trainer init. Replaces 9 device_ptr()/device_ptr_mut() calls (each
triggers cudarc event wait+record → cuStreamSynchronize) with direct
u64 reads from the cached struct.
Affected call sites:
- ISV staging DtoD (step_synthetic)
- Batched loss readback: pi/Q/V/FRD DtoD copies (step_synthetic)
- h_t→h_tp1 encoder DtoD copy (step_with_lobsim)
- trade_context/multires_output ptr pass to perception encoder
All pointers are stable (buffers pre-allocated at init, never
reallocated). Utility functions (read_slice_d, write_slice_f32_d, etc.)
retain device_ptr() as they operate on arbitrary caller-provided buffers
and each already does its own stream sync.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All unconditional. No feature flags. ISV controls magnitudes.
PopArt normalize replaces apply_reward_scale in step_with_lobsim,
with V-correct on v_pred_d and v_pred_tp1_d. Spectral norm runs
one power iteration per step on DQN/IQN/policy weight matrices
after each Adam step in dqn_replay_step. Spectral decouple adds
lambda*mean(logits^2) penalty to Q and pi loss in step_synthetic.
K=3 outcome classifier forward+CE loss runs each step; labels
assigned from reward/done signals. Q-bias correction tracks
mean(Q - return) and emits correction to ISV. Per-branch LR
controller adjusts per-head LR scale factors from loss deltas.
ISV slots 553-572 bootstrapped. Per-branch LR kernel ABI fixed:
current losses passed as args (eliminates slot 572-575 collision
with RL_OUTCOME_AUX_LAMBDA_INDEX).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 3 read_slice_d calls (each allocs+frees MappedF32Buffer) and
4 read_scalar_d calls with pre-allocated persistent isv_staging,
frd_loss_staging, and scalar_staging buffers. Deletes the now-unused
read_scalar_d function.
Eliminates ~20 cuMemHostAlloc + ~20 cuMemFree per step from the hot
path. GPU completes all kernels in 160us/step — host overhead was 14ms.
Target: <1ms host overhead at b=256.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backward HER: on trade close, injects synthetic with peak PnL if
peak > 1.5× actual. Forward HER: evaluates closed trades after 50
steps, injects if holding would have been better.
ISV bootstrap: threshold=1.5, priority_boost=3.0, lookahead=50.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rl_hindsight_track: per-step mid accumulation + peak tracking.
rl_hindsight_inject: backward HER on done, prefix-sum replay write.
rl_hindsight_forward: forward HER after lookahead, prefix-sum write.
No atomicAdd.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PER sample, update_priority, and tree_rebuild run on a dedicated
train_stream. The env-step pipeline (encoder through push_flush)
runs on the main stream. Sync points:
1. Top of step_with_lobsim: train_stream.synchronize() ensures
previous step's priority/rebuild completed before push touches
the replay buffer.
2. Before K-loop: stream.synchronize() ensures push_flush is done
before train_stream's rl_per_sample reads the replay buffer.
3. After rl_per_sample: train_stream.synchronize() ensures sampled_*
buffers are written before step_synthetic/dqn_replay_step reads
them on the main stream.
4. After dqn_replay_step (k>0 path): stream.synchronize() ensures
td_per_sample_d is written before train_stream's priority update.
(step_synthetic already syncs internally.)
The last K-iter's priority update + tree rebuild are NOT synced at
step end — they overlap with the next step's encoder forward on the
main stream, hiding ~1ms of PER maintenance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace write_slice_i32_d_pub (sync + alloc + DtoD) with persistent
MappedI32Buffer staging. Host writes to host_ptr, async DtoD to
frd_labels_d on the train stream — zero sync, zero alloc per step.
Also: make stream field pub for external DtoD access.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace 3 sequential 128-float copy loops with 32 float4 vectorized
loads each. Reduces per-thread memory transactions 4× for the heaviest
kernel (46% of GPU time at 123μs/call → expected ~30μs).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Expanded DiagStaging with 8 more fields (position_lots, pyramid_count,
unit_entry_price, unit_entry_step, unit_lots, unit_trail,
close_unit_index, frd_logits). Eliminates 36 stream syncs per step
that consumed 90.9% of CUDA API time per nsys profile. unit_active
derived from unit_lots != 0 on host (avoids u8-to-f32 DtoD mismatch).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
