- gpu_replay_buffer.rs: orphaned test functions (test_creation, test_beta,
test_clear) and make_stream helper were at module level without #[cfg(test)]
mod tests wrapper. Added proper module boundary.
- gpu_residency.rs, training_stability.rs: sample_proportional() called for
side effect (populates internal buffers asserted on next line). Dropped
unused batch/batch2 bindings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New test coverage for 3 previously untested GPU training paths:
- hyperopt.rs: test_hyperopt_preloaded_data, test_hyperopt_shared_data,
test_hyperopt_paths_consistent — exercises train_with_preloaded_data and
train_with_shared_data which bypass walk-forward (direct train_with_data_full_loop)
- walk_forward.rs: test_walk_forward_multi_fold — tight fractions (0.3/0.1/0.1/0.1)
to guarantee 2+ folds, validating reset_for_fold, graph_aux invalidation, and
CUDA graph survival across fold boundaries
- regression.rs: test_no_hang_single_epoch (VRAM oversubscription guard),
test_counterfactual_experiences_in_buffer (silent data loss guard),
test_gpu_n_episodes_config_honored (auto-scaling removal guard)
Also fixes: unclosed for-loop brace in gpu_per_integration_test.rs (pre-existing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminates all per-fold CPU waste in walk-forward training:
- FxCacheData replaces OHLCVBar as data backbone (features[42] + targets[4] + OFI[8])
- Walk-forward generates index ranges from timestamps, no bar cloning
- DQNTrainer created once, reused across folds via reset_for_fold
- Data uploaded to GPU once via init_from_fxcache, sliced by index per fold
- Trainer accepts &[[f64;42]] + &[[f64;4]] slices, zero Vec<f64> allocation
- PPO uses train_from_slices, no per-fold feature re-extraction
- Ensemble trainers pre-created before fold loop, no per-fold re-upload
- Deleted: prepare_fold_data, FoldData, features_to_trainer_format,
train_ppo_fold, double-buffer, prefetch thread (~500 lines removed)
Smoketest: 160s -> 4.97s (32x speedup)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix 1 (HIGH): Ensemble trainers were created + data uploaded inside the
fold loop, causing redundant GPU uploads per fold per ensemble member.
Moved creation + init_from_fxcache before the fold loop; inside the loop
we now reuse trainers via set_training_range + reset_for_fold.
Fix 2 (MEDIUM): reset_for_fold cleared gpu_data = None, forcing re-upload
on every fold even though the full dataset was already GPU-resident via
init_from_fxcache. Removed the clearing — data stays on GPU across folds.
Fix 3 (LOW): train_fold_from_slices allocated a Vec<f64> per bar via
t.to_vec(). Added train_with_data_full_loop_slices that accepts
&[([f64; 42], [f64; 4])] — both are Copy stack types, zero heap alloc
per element. Added matching collect_gpu_experiences_slices and
run_training_steps_slices helpers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Guard already checks !sharpe_history.is_empty() but unwrap_or makes
the fallback explicit and silences the pre-commit hook warning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three bugs causing the baseline RL training to hang on first epoch while
smoketest completes in 1.2s:
1. VRAM oversubscription (hang root cause): detect_gpu_hardware auto-scaling
computed n_episodes without accounting for 2x counterfactual doubling or
dtod_clone allocations, causing 4x actual memory vs budget. Replaced with
configurable gpu_n_episodes field (smoketest=32, localdev=128, prod=4096).
2. Counterfactual experiences silently dropped: build_next_states_f32 received
n_episodes instead of n_episodes*2, and PER insert used base count instead
of doubled count — ~50% of augmented training data was generated then lost.
3. CudaEvent leak in hot loop: record_event(None) created+destroyed 8000 events
per epoch in forward_online_raw/f32. Pre-allocated 4 events in CublasForward
struct, eliminating driver overhead and handle leaks during CUDA Graph capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add PpoTrainer::train_from_slices(&[[f64; 42]]) that converts to
Vec<Vec<f32>> internally (PPO train() API requires ownership). Replace
the PPO fold loop in train_baseline_rl.rs: eliminates the OHLCVBar
construction shim and calls train_from_slices directly on fxcache
feature slices. Delete the standalone train_ppo_fold function entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds DQNTrainer::train_fold_from_slices(&[[f64;42]], &[[f64;4]]) so the
fold loop in train_baseline_rl passes raw fxcache slices directly, without
the caller constructing any Vec<(FeatureVector, Vec<f64>)>. Removes
features_to_trainer_format_fast helper (no longer needed) and updates
train_dqn_fold to call the new method.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rewrites the train_baseline_rl fold loop to eliminate per-fold waste:
- Data loading: fxcache -> fold index ranges from timestamps (no bar
reconstruction). DBN fallback builds FxCacheData in-place.
- DQN trainer created ONCE before fold loop, fxcache uploaded to GPU
ONCE via init_from_fxcache. Each fold uses set_training_range +
set_val_data_from_slices + reset_for_fold instead of recreating.
- Tokio runtime created ONCE (not per fold).
- Hyperparams construction extracted to build_dqn_hyperparams().
- Deleted: prepare_fold_data, FoldData type, prefetch thread,
DoubleBufferedLoader GPU staging, features_to_trainer_format (old).
- Added: generate_walk_forward_indices_from_timestamps (i64 ns
timestamps, O(log n) partition_point, no OHLCVBar dependency).
- Added: features_to_trainer_format_fast (fxcache targets directly).
- PPO compatibility preserved: constructs minimal OHLCVBars from
fxcache targets for train_ppo_fold (Task 6 will refactor).
- Ensemble mode preserved with per-member trainers for k>0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds fold_train_start/end and fold_val_start/end fields for index-bounded
training. set_training_range() sets the current fold's data range.
Experience collector will use these to index GPU-resident arrays.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clears replay buffer, epoch counters, loss history.
Keeps CUDA context, compiled kernels, CUDA graphs, cuBLAS handles.
FusedTrainingCtx invalidates graph_aux (re-captured on first step).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds index-based walk-forward that returns (start, end) ranges into
pre-loaded arrays instead of cloning bars into Vec<OHLCVBar> per fold.
Uses partition_point for O(log n) date boundary lookups.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Was 0 (auto) which bypassed profile and hit broken AutoBatchSizer.
All configs now have explicit tested batch_size values.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GpuBatch now stores raw u64 device pointers to pre-allocated replay
buffer memory. Eliminates per-step:
- 14 cuMemAlloc calls (7 in sample_proportional + 7 in into_gpu_batch)
- 14 DtoD copies (clone into owned CudaSlice)
- CPU staging Vec and flush() dead code path
Also removed: GpuBatchSlices, into_gpu_batch, dtod_clone_* helpers,
StagedGpuBuffer.staging field, CPU add/add_batch for GpuPrioritized.
update_priorities_cuda now takes u64 raw pointer.
HER relabel_batch_with_strategy takes u64 episode_ids_ptr.
Cold-path Q-value estimation uses compute_q_stats_internal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The old PREFETCH_K loop pre-allocated all batches into Vec<BatchSample>
before training any of them. With 488 steps this meant 976 GPU buffer
lock/sample/alloc cycles upfront, causing multi-minute stalls on H100.
New loop: sample 1 batch from GPU PER, train it, sample next. Zero
prefetch, zero Vec accumulation, natural CPU/GPU interleaving.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The cache key hashed filename + size + mtime. PVC remounts on different
nodes change file timestamps without changing data, producing a different
key every pod boot. Result: fxcache MISS on every H100 run, forcing
10+ min DBN feature extraction from 148GB raw MBP-10 data.
Fix: hash filename + size only. Deterministic across remounts.
Note: existing cache must be rebuilt once (key format changed).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Print fxcache lookup key to diagnose cache miss on H100
- PREFETCH_K=16 (was usize::MAX causing 8M PER samples in one shot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
With max_training_steps_per_epoch removed, num_steps = 4M/8192 = 488.
PREFETCH_K=usize::MAX pre-sampled ALL 488 batches × 8192 × 2 (vaccine)
= 8M sum tree traversals from 24.7M buffer under one CPU lock. Minutes.
Fix: PREFETCH_K=16 — sample 16 batches at a time.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust's std::io::Stdout uses internal BufWriter that bypasses libc,
making stdbuf -oL useless. stderr is unbuffered by default in Rust,
so tracing output appears immediately in container logs.
Also reverts n_episodes debug cap back to 16384.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rust tracing with JSON subscriber in a container without TTY uses
fully-buffered stdout — logs only flush on buffer full or process exit.
This made H100 training appear stuck for 28+ minutes with no output.
stdbuf -oL forces line buffering so epoch logs appear immediately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AutoBatchSizer computes raw VRAM ceiling which can be millions on 80GB.
Cap at 8192: saturates H100 132 SMs for our GEMM sizes (80x256x128),
diminishing returns above this for DQN gradient quality.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced -> (U+2192), x (U+00D7), <= (U+2264), +/- (U+00B1), -- (U+2014)
with their ASCII equivalents. These multi-byte UTF-8 characters caused
the Edit tool to crash consistently on this file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Epoch duration self-balances: bigger GPU → bigger auto-scaled batch →
fewer steps per epoch. The manual cap created 7 different values
(0, 8, 64, 100, 200, 300, 2000) across configs/tests/examples, making
behavior inconsistent between environments.
Removed from: DQNHyperparameters, training profiles (smoketest,
localdev, production), CLI args, Argo templates, hyperopt adapter,
all test overrides, supervised example.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VRAM-derived parameters (batch_size, gpu_n_episodes, buffer_size)
are auto-scaled — CLI overrides bypass this and cause inconsistent
behavior between local testing and production.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
gpu_n_episodes was manually overridden in GPU profiles, training configs,
test files, and hyperopt — all set to 0 or small fixed values that
bypassed the auto-scaling logic, causing a div-by-zero crash in
train_baseline_rl.
Now: single auto-scaling path via optimal_n_episodes() from VRAM/SM
count. No manual override field. Cap at 16384 (consistent with
AutoBatchSizer's 8192 cap pattern). Floor at 32 for small GPUs.
Removed gpu_n_episodes from:
- DQNHyperparameters, PpoHyperparameters structs
- All 4 GPU profiles (rtx3050, h100, a100, default)
- Training profiles (smoketest, localdev)
- ExperienceProfile struct + serde
- Hyperopt adapter
- All test overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GPU monitoring download moved before log_epoch_end so the Epoch complete
line shows the current epoch's per-trade mean reward instead of 0.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sparse trade-completion rewards (98% of bars = 0.0) made mean_reward
always ~0.0, hiding the actual learning signal. Now the GPU monitoring
kernel only accumulates non-zero rewards, giving meaningful per-trade
mean, std, sharpe, and trade count in the epoch summary log.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix EMA kernel: add tau_buf device field, async HtoD via stable host
address, pass pointer not scalar (was ILLEGAL_ADDRESS every step)
- Fix HER relabel kernel: revert indirect ptr_buf to direct bf16 pointer
- Fix PER update kernel: revert indirect ptr_buf to direct u32/bf16 pointers
- Remove IQL per-step DtoH readback (cuStreamSynchronize blocks graph capture)
- Permanently disable cudarc event tracking (SyncOnDrop safe for capture)
- EventTrackingGuard no longer re-enables tracking on drop
- Pre-allocate pass1_event/pass3_event (no cuEventCreate per step)
- RawCudaGraph: raw CUDA driver API bypassing cudarc bind_to_thread
- graph_aux captures ~30 aux kernel launches (HER+clip+EMA+attn+IQL+IQN+CQL)
into single CUDA graph, replayed from step 3+ for zero launch overhead
- IQN/GpuDqnTrainer: tau_host stable field for graph-captured HtoD
- Remove 3 dead indirect pointer kernels from dqn_utility_kernels.cu
- Local RTX 3050: 7.5ms/step steady state (batch=64, 200 steps/epoch)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ROOT CAUSE: GpuDqnTrainer uses double_dqn_stream for Double DQN
target-on-next forward pass. When capturing aux graphs (attention,
IQL, IQN) AFTER graph_forward replay, the captured graph has
dependencies on the double_dqn_stream work → STREAM_CAPTURE_ISOLATION.
The old separate graph_forward/graph_adam captured both streams
properly. The aux graph captures run AFTER replay and inherit
uncapturable cross-stream dependencies.
FIX NEEDED: mega-graph must be captured INSIDE GpuDqnTrainer's
capture_training_graphs where double_dqn_stream is accessible.
All aux ops must be part of the same capture scope.
This commit:
- Disabled cudarc event tracking permanently at FusedTrainingCtx init
- Removed all enable/disable/check_err from capture blocks
- Removed all EventTrackingGuard from gpu_dqn_trainer per-step methods
- Attention device_ptr → raw_ptr for DtoD + memset
- Raw cuStreamEndCapture fallback to prevent stream stuck in capture
- Error logging in attention capture to identify the failing op
- Removed PER update from graph_adam (cross-graph dependency)
- Removed indirect upload + spectral norm from graph_forward capture
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Vaccine:
- Replaced upload_batch_gpu (sync DtoD) with upload_batch_ptrs_6 +
submit_indirect_upload_ops (async indirect kernels). Zero sync.
- Deleted upload_batch_gpu entirely — no callers remain.
Causal intervention:
- Removed entirely from hot path. Was running 14 cuBLAS forward passes
every 100 steps with no readback and no training decision based on
the result. Pure GPU waste.
- Kernel still exists in cubin for future offline analysis.
Causal readback:
- Removed stream.synchronize() + memcpy_dtoh from run_causal_intervention.
Return value was already discarded by caller. Now returns 0.0 immediately.
Sensitivity stays on GPU.
Dead code:
- Deleted upload_batch_gpu (72 lines) — replaced by indirect upload.
Per-step hot path on step 2+:
9 graph replays (~45µs)
5 async HtoD (92 bytes, ~5µs)
Zero sync. Zero DtoH. Zero alloc. Zero CPU compute.
Zero concerns.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN graph now captures the FULL pipeline in one graph:
- decode_actions + fwd/loss + backward + grad_norm + Adam
- trunk gradient (cuBLAS backward into shared weights)
- target EMA (tau from GPU-resident tau_buf, async HtoD before replay)
- IQN→PER loss DtoD copy
IQN EMA kernel changed: float tau → const float* tau_buf (device read).
tau_buf added to GpuIqnHead with async cuMemcpyHtoDAsync per step.
This was the last scalar parameter preventing full graph capture.
regime_scale_td_errors moved into graph_adam submit sequence.
Runs after Adam unflatten, before PER priority update.
Per-step: 7 graph replays + ~9 ungraphed ops
Ungraphed ops (genuinely can't be graphed — batch ptrs change):
- upload_batch_gpu: 6 DtoD + 2 pad_states (batch-specific pointers)
- HER relabel: 1-2 kernels (donor from batch next_states)
- PER priority update: 1 kernel (indices from batch)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
EMA:
- tau now uploaded via async cuMemcpyHtoDAsync_v2 to tau_buf
- EMA kernel reads tau from device buffer (const float* tau_buf)
- No scalar parameter that gets baked at graph capture time
Spectral norm:
- All 10 device_ptr() calls replaced with raw_ptr()
- Eliminates cudarc event recording on weight buffer access
- All parameters (sigma_max, dimensions) are config constants
Both operations are now fully graph-capture compatible:
- Stable buffer addresses
- No cudarc event recording
- No CPU scalar parameters that change between steps (except tau,
which is now GPU-resident)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed all 3 auxiliary Adam kernels to read adam_t from a device
buffer (const int* t_buf) instead of a scalar parameter. This is
required for CUDA Graph capture — scalar params get baked at capture
time and can't change between replays.
CUDA kernel changes:
- iqn_adam_kernel: int adam_t → const int* adam_t_buf
- iql_adam_kernel: int t → const int* t_buf
- attn_adam_kernel: int adam_t → const int* adam_t_buf
Rust changes:
- GpuIqnHead: added t_buf CudaSlice<i32>, async cuMemcpyHtoDAsync
- GpuIqlTrainer: added t_buf CudaSlice<i32>, async cuMemcpyHtoDAsync
- GpuAttention: added t_buf CudaSlice<i32>, async cuMemcpyHtoDAsync
- GpuAttention::adam_step: replaced device_ptr() with raw_ptr()
(eliminates cudarc event recording)
All auxiliary heads are now fully graph-capture compatible:
- Zero CPU scalar parameters that change between steps
- Zero cudarc device_ptr() calls (no event recording)
- All buffer addresses stable (pre-allocated in constructors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
