Implements backtest_gather_kernel.cu (gather_states) which reads directly
from the pre-uploaded features buffer and live portfolio state on GPU,
eliminating the large GPU→CPU download of the full features buffer that
the old gather_states() path performed each step (n_windows×max_len×feat_dim
floats, e.g. 134 MB for 8 windows × 100k steps × 42 features).
The new path: kernel writes [n_windows, state_dim] into states_buf, then
only that tiny buffer (~1.5 KB for 8×48) is downloaded to create the
Candle tensor — a ~100,000x reduction in per-step data transfer.
Wiring changes in GpuBacktestEvaluator:
- Added GATHER_PTX OnceLock + compile_gather_ptx()
- Added gather_kernel (CudaFunction) and states_buf (CudaSlice<f32>) fields
- Added portfolio_dim field (always 3, validated in gather_states())
- Allocates states_buf = n_windows * (feature_dim + 3) in new()
- gather_states() now launches kernel then downloads small output buffer
- metrics download updated to 10 floats/window (Task 13 extended metrics)
- Added 3 new tests: gather PTX compilation, portfolio_dim validation,
state_dim calculation; all 10 gpu_backtest tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GPU-accelerated backtest path that runs walk-forward evaluation
entirely on GPU (env step kernel + metrics reduction), falling back
to the existing CPU path if CUDA is unavailable or the GPU path fails.
Changes:
- Make DQN::q_values_for_batch() public for external Q-value access
- Add RegimeConditionalDQN::batch_q_values() for regime-routed Q-values
- Add DQNAgentType::batch_q_values() dispatch method
- Add DQNTrainer::evaluate_gpu() method (#[cfg(feature = "cuda")])
- Wire GPU-first backtest at the decision point with CPU fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the GPU backtest evaluator (Task 9): uploads walk-forward
window data once, runs the step loop with Candle forward pass + env
kernel, then a single metrics reduction kernel with one final download.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compute_epoch_q_diagnostics now uses compute_q_diagnostics_gpu() free
function with Candle tensor ops. Batches gap stats + per-action means
into single 8-float readback instead of full N×5 to_vec2 download.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One block per window. Parallel reduction for Sharpe, Sortino,
total PnL, max drawdown, win rate, trade count. Single kernel
launch reduces all windows simultaneously.
One thread per walk-forward window, parallel across all windows.
Handles: action→exposure mapping, trade execution with tx costs,
mark-to-market, step return calculation, drawdown tracking.
Portfolio state persists across steps in GPU global memory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DSR portfolio reset and normalizer reset now happen via kernel flags
instead of CPU state mutation. Eliminates cudaStreamSynchronize at
epoch boundaries.
Also fix brace mismatch in gpu_training_guard.rs that left the
accumulate_q_value/read_q_accumulator/reset_q_accumulator methods
outside the impl block (introduced by Task 4 in-progress work).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates N*8 bytes of memcpy_dtoh per experience kernel launch.
MonitoringReducer accumulates stats on GPU, single 48-byte download
at epoch boundary. Zero cudaStreamSynchronize during experience collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add n=0 early-return guard to reduce() to prevent CUDA divide-by-zero
- Add i32::MAX bounds check before casting n to prevent overflow
- Cache PTX compilation with OnceLock<Result<Ptx, String>> (compile once per process)
- Add Safety comment to unsafe block documenting slice/buffer invariants
- Fix doc comment: clarify 48-byte is GPU transfer size (12 × f32)
- Add shmem comment explaining kernel uses static __shared__ only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces per-launch rewards/actions download with single epoch-end
reduction. monitoring_reduce kernel computes mean, std, min, max,
Sharpe estimate, and per-action counts via parallel reduction.
Single 48-byte download instead of N*8 bytes per kernel launch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Episode-boundary resets previously used cold hardcoded constants (0.0f,
1e-8f) for DSR accumulators and EMA normalizer. Now warm-starts from
the persistent epoch state so all episodes within an epoch benefit from
accumulated statistics, not just the first.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The epoch_vol_ema, epoch_dsr_mean, and epoch_dsr_var variables were loaded
from global memory at kernel start but never wired into the actual computation.
The EMA normalizer (ema_mean/ema_var) and DSR accumulators (dsr_A/dsr_B) were
initialized with hardcoded constants, so epoch state round-tripped unchanged.
Changes:
- Seed ema_mean/ema_var from epoch_dsr_mean/epoch_dsr_var at kernel start
- Seed dsr_A/dsr_B from epoch_dsr_mean/epoch_dsr_var at kernel start
- Seed ema_init/dsr_initialized from epoch_step_count > 0 (skip cold start
on subsequent epochs)
- Add local_vol_ema/local_median_vol seeded from epoch_vol_ema/epoch_median_vol
- Update vol EMA each timestep from market feature index 3 (log close return),
mirroring CPU DQNTrainer::vol_ema / median_vol logic
- At writeback, write actual computed dsr_A/dsr_B or ema_mean/ema_var (conditioned
on use_dsr), and computed local_vol_ema/local_median_vol, instead of unmodified
loaded epoch values
- Applied identically to both dqn_full_experience_kernel (standard) and
dqn_full_experience_kernel_warp (warp-cooperative) variants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU PER replay buffer had a hardcoded 4 GB MAX_BYTES limit that
rejected the auto-sizer's 10M-entry proposal on H100 (80 GB VRAM).
Now per_max_buffer_bytes() computes 20% of total VRAM (min 1 GB) and
flows through OptimalReplayConfig → DQNConfig → GpuReplayBufferConfig
so both subsystems agree on the budget.
Also fixes misleading regime detection log (indices 211/219 → 40/41)
and renames dqn_config_2025 → dqn_default_config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates cudaStreamSynchronize between epochs by keeping vol EMA,
portfolio state, and DSR normalizer in persistent CudaSlice buffers.
Kernel reads initial state at launch, writes final state at exit.
- Add epoch_state CudaSlice<f32>[8] field and reset_flags u32 bitfield
to GpuExperienceCollector struct
- Allocate epoch_state in new() with sensible defaults (vol_ema=0.01,
initial_capital for portfolio, dsr_var=1.0 to avoid div-by-zero)
- Pass epoch_state and reset_flags as final args to both kernel variants
(standard per-thread and warp-cooperative)
- Kernel: thread/lane 0 of block 0 applies reset flags atomically with
__threadfence, all threads read 8-float epoch state from L1-cached
global memory, last block writes back updated values at exit
- Auto-clear reset_flags after each launch (one-shot semantics)
- Add set_reset_flags(), clear_reset_flags(), epoch_state_gpu(),
rewards_gpu(), actions_gpu() public methods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In candle-core (git 671de1d), min(D) and max(D) return Result<Tensor>,
not Result<(Tensor, Tensor)>. Use flatten_all() then min(0)/max(0) for
scalar reduction instead of the tuple destructuring pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Task 14: evaluate_baseline is at crates/ml/examples/evaluate_baseline.rs
(not bin/fxt/src/commands/), fix cargo check command and git add path
- Task 15: process_bar_factored takes (usize, &OHLCVBarF32, &FactoredAction)
not (f64, usize, f64, f64), fix test to use correct types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training data is already on the PVC — the rclone sync step was
wasting 2-3 minutes per job. Also fixes --output-dir → --output
to match the actual CLI flag.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
15-task plan covering Phase 1 (seal GPU→CPU roundtrips in training loop),
Phase 2 (vectorized GPU backtest environment for hyperopt), and Phase 3
(general GPU backtester integration).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three-phase plan to eliminate all GPU→CPU roundtrips from training:
- Phase 1: Seal training loop (persistent GPU epoch state, async monitoring)
- Phase 2: Vectorized CUDA backtest kernel for hyperopt evaluation
- Phase 3: General-purpose GPU backtester replacing CPU SIMD path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: `option_env!("FOXHUNT_BUILD_VERSION")` in common/build_info.rs
was a compile-time macro tracked by cargo fingerprints (Rust 1.80+). Every
pipeline run with a new tag invalidated `common` → cascading rebuild of all
38 dependent workspace crates, even when zero source files changed.
Fix: replace `option_env!()` with `std::env::var()` (runtime LazyLock). Cargo
no longer tracks the version env var, so `common` only recompiles when its
source actually changes.
Also: skip git checkout when HEAD already matches target SHA (zero mtime
changes), and drop `-x` from git clean to preserve gitignored files.
Expected: ~3.7min → <1min for unchanged-crate rebuilds on warm PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU-accelerated Branching DQN training:
1. **GPU experience collector**: NoisyLinear creates standalone Vars via
Var::from_tensor(), bypassing VarMap registration. The GPU collector
looks up weights by name ("value_fc.weight") from VarMap and falls
back to CPU (~5x slower) when missing. Fix: register mu vars in
VarMap at construction, keep sigma vars standalone.
2. **Optimizer device mismatch**: Using only vars().all_vars() left
NoisyLinear head params frozen. backward() produces gradients the
optimizer doesn't know about → device mismatch in clip_grad_norm.
Fix: all_trainable_vars() = VarMap (shared+mu) + sigma.
3. **Single-threaded CPU bottleneck**: Runtime::new() creates a
current-thread scheduler → 1 OS thread → all async work serialized.
Fix: multi-thread runtime (4 workers) created once in DQNTrainer::new(),
shared across preload/training/backtest phases. Eliminates 3 fallback
Runtime::new() callsites.
Also: polyak_update_var_pairs with debug_assert_eq, two-phase target
network sync (VarMap Polyak + sigma var_pairs Polyak), copy_weights_from
handles NoisyLinear heads.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep gpu-warmup only in compile-and-train-template where it runs
parallel with the CPU compile step. The other templates don't need
it — CI pipeline doesn't always train, and training-workflow already
has the GPU node available from the triggering event.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
git clone gives all files the current timestamp, causing cargo to
recompile every workspace crate even when source is unchanged. Switch
to persistent checkout on the PVC: git fetch + git checkout --force
only updates files that actually changed between commits, preserving
mtimes on unchanged files so cargo correctly skips them.
- compile-services: /cargo-target/src persistent checkout
- compile-training: /cargo-target/src persistent checkout
- compile-and-train: /cargo-target/src persistent checkout + CARGO_HOME
- First run: full clone (fallback), subsequent: fetch + checkout
Expected: ~40 crate recompile → only changed crates (~1-2 min)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 37+ workspace crates, cargo's incremental fingerprinting already
skips unchanged crates — sccache adds per-rustc wrapping overhead for
no benefit. The real speedup is CARGO_HOME on PVC (eliminates crate
re-downloads every build).
Kept: CARGO_HOME=/cargo-target/cargo-home on PVC
Removed: RUSTC_WRAPPER=sccache, SCCACHE_DIR, CARGO_INCREMENTAL=0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both compile-services and compile-training were doing full rebuilds
every run because:
1. No RUSTC_WRAPPER=sccache — sccache was installed but never used
2. No persistent CARGO_HOME — crates.io registry re-downloaded each time
3. Incremental compilation doesn't work across git clones (mtime-based)
Fix: enable sccache (content-hash based, clone-agnostic), persist
CARGO_HOME on the PVC, disable incremental (conflicts with sccache).
Expected: first run populates cache, subsequent runs ~2-3min vs ~8min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates failed because ci-deploy role was missing
permissions for eventbus (argoproj.io), services (core), and
roles/rolebindings (rbac) needed to apply events/*.yaml files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo forbids mixing `dependencies` and `depends` in the same DAG.
Converted all tasks to `depends` with explicit state conditions so
infra steps can wait for image rebuilds while still running when
rebuilds are skipped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uses Argo `depends` conditions so apply-argo-templates and
terragrunt-apply wait for rebuild-ci-builder-cpu to finish (or skip)
before running, ensuring they always get the latest image.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates needs git (clone repo) + kubectl (apply manifests).
ci-builder-cpu has git but lacked kubectl — added it. foxhunt-runtime
has kubectl but no git and runs as non-root, so can't be used here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline now auto-applies WorkflowTemplates, EventSources, Sensors,
and RBAC when infra/k8s/argo/ files change — no more manual kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace fragile `git fetch --depth=1 origin "$SHA"` (fails with short SHAs)
with `git clone --no-checkout --filter=blob:none` + `git checkout "$SHA"`
across all 3 Argo templates (5 clone blocks total)
- Upgrade CI compile pool from POP2-32C-128G to POP2-HC-32C-64G
(higher clock EPYC cores, 28% cheaper at €0.851/hr vs €1.18/hr)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache forces CARGO_INCREMENTAL=0, causing all 37 workspace crates to
recompile from scratch every CI run (~20 min). Only upstream deps were
cached (635 hits); 109 workspace rlib crates were non-cacheable.
Changes:
- Add cargo-target-cpu and cargo-target-cuda PVCs (30Gi each)
- Mount persistent target dir at /cargo-target via CARGO_TARGET_DIR
- Drop RUSTC_WRAPPER=sccache and SCCACHE_DIR from both compile steps
- Drop hardcoded CARGO_BUILD_JOBS=14 (let cargo auto-detect from nproc)
- Add 25GB cleanup guard to prevent unbounded PVC growth
- Update binary copy paths to use $CARGO_TARGET_DIR/release/
First build (cold PVC) is same speed. Subsequent builds with typical
3-5 file changes should drop from ~20 min to ~2-3 min via incremental.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three optimizations targeting GPU allocation churn and lock contention:
1. In-place polyak update: Var::set() reuses existing GPU buffer instead
of Var::from_tensor() which allocates a new one per call. Eliminates
~10,000 cudaMalloc/cudaFree per epoch (20 params × 500 steps).
2. Fused affine ops: Replace 13 Tensor::full()/Tensor::ones() constant
tensor allocations per step with tensor.affine(mul, add) — a single
fused kernel. Patterns: 1-x → x.affine(-1,1), γ*x → x.affine(γ,0),
0.5*x² → (x*x).affine(0.5,0). Applied across all 4 loss paths
(branching Bellman/Huber, standard Bellman/Huber, IQN). Eliminates
~6,500 GPU allocs/epoch.
3. Batch pre-sampling (K=8): Sample 8 batches under one READ lock, train
all 8 under one WRITE lock. Reduces async lock acquisitions from
2×N to 2×ceil(N/8). Priority staleness across 8 steps is negligible.
Combined estimated impact: 20-35% H100 throughput improvement.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Candle's Tensor::cumsum(0) internally allocates an [n,n] upper-triangular
matrix (9.3 GB for n=50K) causing OOM on GPUs ≤48 GB. Replace with a
block-parallel Hillis-Steele scan kernel that is O(n) in time and memory.
Additional fixes in this commit:
- Break autograd chain leak in loss/grad accumulation via .detach()
(was leaking ~32 MB/step across entire training run)
- Release features_raw_cuda/targets_raw_cuda after GPU experience
collection (~164 MB VRAM reclaimed)
- Use softmax eval (temp=0.3) in walk-forward backtest to prevent
action collapse causing trades=0 on early-stage models
Validated: 1286 tests pass (408 ml-dqn + 878 ml), 0 failures.
VRAM stable at 754 MB across 45K+ training steps on RTX 3050 Ti.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents experience collector output tensors from starving backward
graph and GPU PER on VRAM-constrained GPUs like RTX 3050 Ti 4GB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
free_memory_mb is sampled after model/optimizer load, so subtract only
backward-graph + fragmentation + data upload headroom (400 MB), not the
full model/PER/data stack. Increase collector share to 50%.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Staging/GPU replay buffer: truncate batch when it exceeds ring buffer
capacity (CPU fallback can flush 100K+ experiences at once)
- GpuExperienceCollector: compute shmem tile rows dynamically to stay
under 48 KB default limit instead of hardcoded 64-row constant
- Auto batch size: subtract concurrent VRAM consumers (~530 MB model +
optimizer + PER + data) before budgeting experience collector at 40%
- Hyperopt DQN adapter: GPU PER always on, include replay buffer in
VRAM estimate for small GPUs (was excluded assuming CPU-only PER)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the cfg gate inside the method body instead of gating the entire
function. Without cuda, the GpuPrioritized variant doesn't exist so
matches! returns false — no need for separate cfg blocks at call sites.
Fixes compile-services CI failure (services build without --features cuda).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN hyperopt adapter: remove static VRAM gate for GPU experience
collector and GPU PER — dynamic scaling handles constraints at
runtime with graceful CPU fallback on init failure
- Remove is_parquet_file branching from hyperopt eval path (all data
loads via DBN pipeline now)
- Update training example CLI args for consistency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
