Replace monolithic compile-all with granular per-binary change detection.
detect-changes now outputs space-separated package/example lists based on
a dependency map from source directories to binary targets:
- Shared crates (common, config, Cargo.toml) → all binaries
- Service-specific dirs → only that service binary
- Domain crates (trading_engine, risk) → dependent service subset
- ML crates → ml-training-service + trading-service + all training
- ML subdirs (trainers/, hyperopt/, evaluation/) → specific training binaries
compile-services and compile-training accept package lists and build only
affected binaries, saving ~20-30s link time per skipped binary.
deploy-services restarts only affected deployments (trading-service
excluded from auto-deploy for safety).
Fix: 'latest' package update now replaces individual files instead of
deleting the entire package, preventing corruption during partial builds.
compile-and-train-template: derive needed training binaries from model
parameter (3 instead of 7), drop unused training_uploader build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate all per-step GPU→CPU synchronization barriers from the
training guard. Replace device+host buffer pairs and memcpy_dtoh
with cuMemHostAlloc(DEVICEMAP) mapped pinned memory.
Key changes:
- MappedBuffer struct: cuMemHostAlloc + cuMemHostGetDevicePointer_v2
allocates memory visible to both CPU and GPU simultaneously
- Double-buffering: kernel writes to buffer[N%2], CPU reads buffer
[(N-1)%2] — one-step delayed halt detection, zero sync
- __threadfence_system() in CUDA kernels ensures writes visible to
CPU across PCIe without explicit memcpy
- read_volatile on host pointer prevents CPU-side caching
Eliminated:
- check_and_accumulate: 28-byte memcpy_dtoh (every training step)
- qvalue_stats: 16-byte memcpy_dtoh (every 50 steps)
- qvalue_divergence: 20-byte memcpy_dtoh (every 50 steps)
Kept: read_accumulators memcpy_dtoh (12 bytes, epoch boundary only —
accumulator buffer stays in device memory for kernel read-modify-write).
1286 tests pass (878 ml + 408 ml-dqn), 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two remaining GPU→CPU synchronization barriers in the CUDA-active
training loop:
1. detect_dead_neurons() → to_vec0() called every training step from
log_diagnostics() in the guard path. Added check_gradient_collapse()
method that performs the same collapse detection logic but skips the
expensive per-parameter weight scan. Dead neuron detection now only
runs at epoch boundary via log_diagnostics().
2. forward() Q-value clipping monitoring → to_vec2() called every 1000
steps during compute_loss_internal() and Q-value estimation forward
passes. Added training_forward_active flag that gates the monitoring
block; set to true during all training-path forward() calls
(compute_loss_internal + Q-value estimation), false during
inference/evaluation.
All 1577 tests pass (878 ml + 408 ml-dqn + 291 ml-core), 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire route_exposure_to_factored() into both select_actions_batch_gpu
and select_actions_batch GPU paths, eliminating per-item CPU routing.
Previously, exposure indices (0-4) were downloaded from GPU and routed
to factored indices (0-44) one-by-one on CPU via route_action(). Now
the exposure→factored mapping runs entirely on GPU via the routing
kernel, with a single batch readback of the final factored indices.
Branching DQN path unchanged (already produces factored indices 0-44).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace CPU-bound Q-value monitoring with GPU-resident qvalue_stats and
qvalue_divergence kernels from GpuTrainingGuard. The two readback sites
(estimate_avg_q_value_with_early_stopping's mean_all().to_scalar() and
log_q_values' to_vec2()) are now bypassed when the GPU training guard is
active. CPU fallback path preserved for non-CUDA builds and guard-absent
scenarios.
Changes:
- Add DQN::log_q_values_from_stats() accepting pre-computed GPU stats
- Add DQNAgentType::log_q_values_from_stats() delegate
- Replace Q-estimation in train_step_single_batch with GPU kernel path
- Replace Q-estimation in train_step_with_accumulation with GPU kernel path
- Import IndexOp trait for Tensor::i() in trainer.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a standalone CUDA kernel that converts DQN exposure indices (0-4)
to factored action indices (0-44) entirely on GPU, reusing the existing
route_order() device function from common_device_functions.cuh. Wired
into GpuActionSelector as route_exposure_to_factored() method, following
the same DtoD copy pattern as the existing select_actions methods. This
eliminates a GPU->CPU->GPU roundtrip when post-hoc routing is needed
after epsilon_greedy_select.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the GPU->CPU sync barrier (to_vec1 readback) in the DQN training
hot path with the GpuTrainingGuard CUDA kernel that performs NaN detection,
loss clipping, and gradient collapse checks entirely on-device. The guard
is lazy-initialized on first training step and falls back to the original
CPU readback path if CUDA kernel compilation fails.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Seven pure-Rust reference tests covering all four training_guard_kernel.cu
kernels: PTX NVRTC smoke-test (skips on CPU-only), NaN/Inf detection,
loss clip, gradient collapse, accumulator averaging, GuardResult boolean
threshold construction, and qvalue_stats_reduce per-sample max-Q reduction.
Zero clippy warnings in the new file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wraps 4 CUDA kernels (training_guard_check, training_guard_accumulate,
qvalue_stats_reduce, qvalue_divergence_check) with a Rust struct that
uses OnceLock PTX caching, pre-allocated device buffers, and host-side
Vec mirrors for zero-allocation readbacks per training step.
Accumulator (3-float acc_buf) stays on-device for epoch-boundary
averaging without CPU roundtrips.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
11-task plan across 4 chunks: Training Guard kernel + wrapper (Tasks 1-3),
wire into trainer (Tasks 4-6), Q-value monitor + action routing (Tasks 7-9),
experience collector audit + final verification (Tasks 10-11).
Eliminates all 10 GPU→CPU sync barriers from the DQN training hot path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Defines architecture for eliminating all 10 GPU→CPU sync barriers from
the training loop via 4 new GPU components: Training Guard (pinned
memory predicates), Q-Value Monitor (on-device accumulator), GPU-resident
action selection, and async experience collector readback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add use_dsr, dsr_eta, and n_steps fields to ExperienceCollectorConfig
with safe defaults (DSR off, eta=0.01, n_steps=1). Pass them as kernel
args after fill_simulation_enabled, matching the parameter order added
in Tasks 4 and 5. Wire from DqnHyperparams in the trainer hot path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirror the DSR (Differential Sharpe Ratio) and n-step return
accumulation added by Task 4 to the scalar kernel into the warp
kernel (dqn_full_experience_kernel_warp). DSR/n-step state runs
on lane 0 only, matching the existing reward computation pattern.
All accumulators are reset on episode boundaries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds Differential Sharpe Ratio reward shaping and n-step return
accumulation to dqn_full_experience_kernel (scalar path). DSR replaces
EMA normalization when use_dsr=1; n-step ring buffer accumulates
discounted returns for effective_n>1. Both are reset on episode
boundaries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Append third CUDA kernel to epsilon_greedy_kernel.cu for Branching DQN:
3 independent epsilon-greedy heads (exposure/5, order/3, urgency/3)
compose to factored action index 0-44 (exposure*9 + order*3 + urgency).
Add Rust composition tests verifying full 45-action coverage and
round-trip decomposition correctness.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add backticks to type names in doc comments (doc_markdown)
- Mark eligible functions as const fn (missing_const_for_fn)
No behavior changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
module_name_repetitions: PpoConfig in ppo module is conventional ML naming.
Renaming breaks every import across the workspace.
integer_division: Basis point calculations, batch size math, combinatorial
formulas — truncation is intentional. Float conversion would introduce bugs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
is_gpu_prioritized() and grad_norm squeeze are only available with CUDA.
Without the gate, compile-services (CPU-only) fails with E0599.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-step to_scalar()/to_vec1() readbacks with GPU-resident
tensor accumulation and single epoch-boundary sync. On H100 this
removes ~12-15 pipeline flushes per training step (~5μs each),
enabling full GPU saturation with zero CPU sync in the inner loop.
Key changes:
- GpuTrainResult: train_step() returns GPU scalar tensors (loss_gpu,
grad_norm_gpu) instead of f32 — zero readback per step
- Regime conditional: train all 3 heads unconditionally with
zero-masked weights (mathematical no-op) instead of 3-6
to_scalar() mask count checks per step
- GPU PER: max_priority as GPU tensor with flush_max_priority(),
delta-based priority update via index_add (no CPU dedup)
- Deferred diagnostics: NaN detection, CQL logging, Q-value
estimation all moved to epoch boundary where pipeline is
already synced
- Legacy CPU-readback wrappers removed entirely
9 files changed, +393/-295 lines. 520 DQN tests passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate GPU→CPU→GPU roundtrip in the experience collection hot path.
Kernel outputs (states, rewards, actions, dones) now stay on GPU via
device-to-device copy into candle Tensors. Only rewards + actions are
downloaded for monitoring metrics (~0.5MB vs ~3.14GB at 32K episodes).
- Extract launch_kernel() helper from collect_experiences() (DRY)
- Add collect_experiences_gpu() — DtoD path for GPU PER
- Add cuda_slice_to_tensor_f32/i32_to_u32 DtoD copy utilities
- GPU next_states via tensor narrow/cat/reshape (no CPU loop)
- Trainer selects GPU vs CPU path based on is_gpu_prioritized()
- stream.synchronize() barrier ensures cross-stream data visibility
0 warnings, 1266 tests pass (874 ml + 392 ml-dqn)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Forward-port from worktree-gpu-hotpath-audit (2 commits):
1. Zero-roundtrip DQN experience collection via cuMemcpyDtoDAsync:
- GpuExperienceCollector uses device-to-device copies for state tensors
- Shared memory weight caching in GPU replay buffer
- NaN priority clamping on GPU (no CPU readback)
2. Eliminate all GPU→CPU roundtrips from training loop:
- GpuTrainResult: loss + grad_norm stay as GPU scalar tensors
- Single to_scalar() readback at epoch boundary (not per step)
- GPU-resident loss accumulation across training steps
- RegimeConditional head selection via GPU tensor ops
- Removed per-step NaN diagnostic checks (now epoch-level)
Also removes dead `states_tensor` field from ComputeLossResult and
fixes redundant field name clippy warning.
Expected: ~15-20% training throughput improvement on H100 by
eliminating synchronous GPU→CPU transfers in the inner loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ubuntu 24.04 minimal doesn't include passwd (useradd/groupadd).
Debian bookworm-slim had it by default. Required for foxhunt user.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cargo defaults to /proc/cpuinfo CPU count (32 host cores) but each
compile pod is cgroup-limited to 14 CPU request. Without this, both
pods spawn 32 threads each (64 total) causing heavy throttling on
the 32-core POP2 node. Setting CARGO_BUILD_JOBS=14 per pod avoids
context-switch overhead and uses cgroup-aware parallelism.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removes temporary benchmark comment. Docker changes from previous commit
will trigger image rebuilds. This run serves as warm-cache benchmark
after cold-cache run completes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Temporary comment to trigger detect-changes for both services and
training compile steps. Will be removed after benchmarking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch sccache backend from S3 (MinIO) to persistent local storage.
Two 20Gi PVCs (sccache-cpu, sccache-cuda) on scw-bssd-retain eliminate
network I/O for cache reads/writes. Both compile steps always run on
the same node so RWO is sufficient.
Removes 13 S3 env vars per compile step, replaces with SCCACHE_DIR=/sccache.
Updates ci-pipeline-template, compile-and-train-template, kustomization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. PER duplicate index accumulation (HIGH): update_priorities_gpu()
used index_add delta trick which accumulates deltas for duplicate
indices, overshooting target priority. Now deduplicates the small
indices tensor (batch_size=256, ~1KB) via HashMap before delta
computation. Fast path (no dupes) reuses original tensors.
2. RegimeConditional silent experience drop (MEDIUM):
insert_batch_tensors() returned Ok(()) for CPU replay buffers,
silently discarding all GPU-collected experiences. Now converts
tensors→Vec<Experience> via extracted helper (lazy, only on first
CPU head encountered) and inserts into CPU buffer.
3. Unsafe direct indexing (LOW): gpu_experience_collector.rs used
&states[s..e] in fallback paths, violating deny(indexing_slicing).
Replaced with .get() safe bounds checking.
392/392 ml-dqn + 874/874 ml tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add warp-cooperative CUDA kernel for DQN experience collection that
exploits H100's improved warp shuffle throughput. On sm_90+, 32 threads
(1 warp) cooperate on each dot product via __shfl_xor_sync butterfly
reduction, cutting per-thread register pressure from ~5.5KB to ~200B
and enabling full occupancy on 132 SMs.
Key additions:
- 6 warp-cooperative device functions in common_device_functions.cuh:
distributed/broadcast matvec (clean + NoisyNet), warp RMSNorm,
warp_reduce_sum_all
- 4 TILE_LAYER_WARP_* macros using __syncwarp() for warp-level sync
- 3 warp forward passes: standard dueling, NoisyNet, C51 distributional
(distributed heavy layers + broadcast atom layers for softmax)
- Full warp kernel: dqn_full_experience_kernel_warp with lane-0
simulation, strided state scatter, warp-shuffle curiosity gather
- Host-side compute capability detection (sm_major >= 9) with
automatic fallback to standard 256-thread kernel on older GPUs
- cuCtxSetLimit(STACK_SIZE, 16KB) for standard kernel safety
874/874 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three zen-validated optimizations for H100 GPU experience collection:
1. Dynamic NOISY_MAX_DIM: inject via NVRTC as max(state_dim, shared_h1,
shared_h2). On H100 with SHARED_H1=512, noise now covers all 512 input
dims instead of truncating at 256. Fixes exploration correctness bug.
2. MAX_EPISODES_LIMIT raised 4096→32768: H100 (132 SMs) can now run up to
32768 episodes per kernel launch = 128 blocks = ~97% SM utilization
(was 16 blocks = 12%). All 3 trainer.rs caps updated to match.
3. Scratch array aliasing: scratch_v and scratch_a now alias scratch1
memory after shared layers complete, saving 1024 bytes per thread.
Per-thread local memory reduced from ~6.5KB to ~5.5KB.
Zen validation confirms: kernel is safe and correct for H100 deployment.
Remaining architectural improvements (warp-cooperative matvec, batched
GEMM) deferred to future optimization phase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
index_add with duplicate indices double-applies deltas — e.g. if index i
is sampled twice with delta -0.5, priority goes 1.0 + (-0.5) + (-0.5) = 0.0,
causing NaN in importance sampling weights. Post-clamp to [epsilon, 1e6]
ensures priorities stay positive.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: SHMEM_MAX_IN_DIM was hardcoded to 256, but on H100 with
hidden_dim_base=2048, SHARED_H1=SHARED_H2=512. The cooperative tile
loader wrote 64×512=32768 floats into shmem allocated for 64×256=16384,
causing CUDA_ERROR_INVALID_VALUE on kernel launch.
Fixes:
- Make SHMEM_MAX_IN_DIM injectable via NVRTC #define (was hardcoded)
- Compute max(state_dim, shared_h1, shared_h2) and inject at compile time
- Use dynamic value for host-side shmem_bytes allocation
- Shrink eps_out[NOISY_MAX_DIM] → eps_out[SHMEM_TILE_ROWS] (saves 768B/thread)
- Force smoke tests to Device::Cpu (CUDA driver sensitivity to binary layout)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
optimal_n_episodes() returns 8192 on H100 (132 SMs) but the GPU
experience collector pre-allocates buffers for MAX_EPISODES_LIMIT=4096.
Exceeding that caused "Episode count exceeds allocated buffer size"
and silent fallback to CPU. Now all 3 auto-scaling sites clamp at 4096.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
