- apply-argo-templates step now also applies argo-workflow-netpol.yaml
so NetworkPolicy changes auto-deploy on push
- Deleted stale compile-training WorkflowTemplate from cluster
(superseded by compile-and-train)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CUBLAS_WORKSPACE_CONFIG and NVIDIA_TF32_OVERRIDE are set once at
startup before any threads or CUDA work begins. The unsafe block is
correct but triggers -W unsafe-code. Adding #[allow(unsafe_code)] at
the call site silences the warning while keeping the safety comment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Image builds are handled by ci-pipeline, not by compile-and-train or
compile-and-deploy. Having them here caused network policy failures
(Kaniko can't reach registry) and securityContext conflicts (Kaniko
needs root). Also removed pod-level securityContext from
compile-and-train since ci-builder needs root for cargo on PVC.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
(GID 1000 already exists in ubuntu:24.04)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-and-train workflow uses component label 'compile-and-train'
which wasn't covered by existing network policies. Pods were blocked by
default-deny-all from reaching GitLab SSH (git clone) and HTTP (package
upload/download). Combines compile egress (SSH, external HTTPS, DNS) with
training egress (Tempo OTLP, GitLab HTTP).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PVCs may retain ownership from a previous pod UID (e.g. after
ci-builder image rebuild). Git 2.35.2+ refuses to operate without
safe.directory set. Applied to both persistent-checkout templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The cargo-target PVC retains the source checkout between workflow runs.
When the container UID differs from the PVC owner, git refuses to operate
with "dubious ownership" error. Adding safe.directory resolves this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No cudnn feature flags remain after 6ba52425 — the devel headers
and libs were dead weight (~1.5 GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- gradient_utils: Add TensorId-based fallback when Var identity mismatch
causes 0/N vars to match GradStore (Candle Adam clones Arcs). Fallback
computes norm AND clips via insert_id. Throttled warning (1st + every
1000th). 7 unit tests including mismatch-actually-clips.
- monitoring: Track full 45-action factored space (5 exposure × 3 order
× 3 urgency). Fix validate_rewards false alarm on GPU path where single
aggregated mean_reward per epoch gives N=1 → std=0.
- trainer: GPU experience collection routes exposure actions through
route_action() for factored tracking instead of exposure-only counts.
Applied in both per-step and epoch-summary paths.
- train_baseline_rl: Auto-detect VRAM <8GB → disable GPU replay buffer
to prevent OOM on RTX 3050 Ti class GPUs.
- smoke_test_real_data: E2E DQN training test with 6 assertions (epoch
completion, loss decrease, finite losses, Q-value divergence, 45-action
space, finite gradient norms).
Validated: 1642 tests pass (ml=915, ml-core=311, ml-dqn=416), 0 clippy
warnings, baseline RL trains 10 epochs on CUDA with Sharpe +5.45.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move // gpu-ok: annotations to same line as violation patterns so the
guard script's grep -v filter actually suppresses them. Fixes 6 false
positives in ensemble adapters (dqn, ppo, liquid, kan, tggn, diffusion).
Replace hardcoded 48KB shmem limit in compile_forward_kernel() with
GPU-aware query (max_shared_memory_kb) — matches gpu_experience_collector
pattern. H100 now gets 128-row tiles (was 64), eliminating tile loops
for ≤128-dim layers in the backtest forward kernel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Capture the full DQN backtest step loop (gather → forward → DtoD →
env_step × max_len) as a replayable CUDA Graph:
- evaluate_dqn_graphed(): captures on first call via
CudaStream::begin_capture/end_capture, replays cached graph on
subsequent calls. Falls back to evaluate_dqn() on any failure.
- invalidate_dqn_graph(): discards cached graph when weights change.
- SendSyncGraph: newtype wrapper for CudaGraph (single-threaded use).
- Full unrolled capture: each step's step_i32 argument is baked in at
capture time, avoiding GPU-resident step counter complexity.
Eliminates ~5-10μs per kernel launch × 4 kernels × max_len steps
of CUDA driver overhead per evaluation.
evaluate_baseline.rs: added --cuda-graphs CLI flag to opt in.
14 backtest evaluator tests pass, 0 clippy errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-stream pipeline (evaluate() closure path):
- Secondary env_stream with CudaEvent sync allows env_step to overlap
with gather/forward of the next iteration on H100's 132 SMs
Pure-CUDA DQN forward (evaluate_dqn):
- backtest_forward_kernel.cu: warp-cooperative dueling Q forward via NVRTC
- Eliminates candle per-op dispatch overhead, enables future CUDA Graph capture
- evaluate_baseline.rs: auto-detects dueling network and uses pure-CUDA path
Pure-CUDA PPO forward (evaluate_ppo):
- backtest_forward_ppo_kernel.cu: actor MLP → softmax → 45→5 exposure collapse → argmax
- One thread per window, bypasses candle entirely
CUDA supervised signal→action (evaluate_supervised):
- backtest_forward_supervised_kernel.cu: threshold bucketing kernel
- Candle still used for model forward (TFT/Mamba/etc), but argmax is GPU-native
Shared helpers: launch_gather, launch_env_step_on, launch_metrics_and_download
deduplicate code across all four evaluation paths.
914 tests pass, 0 clippy errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Lower warp kernel SM threshold from sm_90 to sm_70 (covers all modern GPUs)
- Add q_forward_branching_warp_shmem: warp-cooperative branching DQN forward
with 3 independent advantage heads + shared memory tiling
- Fix online/target dispatch to use warp branching variant when available
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore 45-action factored space via Branching DQN (Tavakoli 2018),
outputting 11 Q-values (5+3+3) instead of 45. This was reduced to 5
exposure-only actions during debugging and was never intended as permanent.
- Enable use_branching: true by default in DQNConfig and DQNHyperparameters
- Add branching paths to select_action_with_confidence and select_action_inference
- Update agent.rs select_action_factored for branching-aware selection
- Expand CountBonus to per-branch tracking with bonuses_branched()
- Add order_type + urgency distribution tracking in monitoring
- Add DQN_ORDER_ACTIONS=3, DQN_URGENCY_ACTIONS=3, DQN_TOTAL_ACTIONS=45 to CUDA header
- Fix 7 pre-existing clippy doc_markdown errors in regime_conditional.rs
- Fix pre-existing cognitive_complexity in replay_buffer_type.rs (extract helpers)
- Fix flaky GPU test OOM under parallel execution (CPU fallback + test VRAM safety)
- Delete unused flash_attention submodules (block_sparse, causal_masking, etc.)
- Add GPU hot-path guard scripts and ensemble/hyperopt adapter improvements
Tests: ml-dqn 416/0, ml 905/0, clippy 0 errors on both crates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Continuous vectors and expected counts updated from 12 to 14 params
after adding signal_high_bps and signal_low_bps to Mamba2Params.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- evaluate_supervised_fold_gpu(): loads any of 8 supervised models via
UnifiedTrainable, runs GPU backtest with signal threshold mapping
- TFT uses tft_quantile_to_signal() to extract median quantile before
threshold comparison; scalar models use signal_to_action_scores() directly
- create_supervised_model() factory + find_supervised_checkpoint() helper
- Main loop dispatches GPU-first for all supervised models when --gpu-eval
- RefCell wrapper bridges &mut self forward() into Fn(&Tensor) closure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All supervised hyperopt adapters (TFT, Mamba2, Liquid, KAN, xLSTM, TGGN,
TLOB, Diffusion) now include:
- signal_high_bps and signal_low_bps in ParameterSpace (tunable thresholds)
- backtest_sharpe and backtest_trades fields in Metrics structs
- extract_objective prefers GPU backtest fitness over val_loss when available
- All tests updated (101 passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- evaluate_ppo_fold_gpu(): loads PPO checkpoint, runs GPU backtest with
ppo_to_exposure_scores (45→5 collapse), uses GpuBacktestEvaluator
- PPO main loop: tries GPU path first when --gpu-eval (default), falls
back to CPU if CUDA unavailable or error
- PPOMetrics: backtest_sharpe/backtest_trades fields for GPU walk-forward
- PPOTrainer::run_gpu_backtest(): runs backtest on validation 20% split
- extract_objective(): prefers backtest_fitness when GPU results available,
falls back to -avg_episode_reward on non-CUDA builds
- 2 new tests: backtest fitness priority + few-trades penalty
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire GpuBacktestEvaluator into PPO hyperopt trials so the optimizer
ranks candidates by walk-forward Sharpe ratio instead of raw episode
reward. The PPO actor's 45-action softmax probabilities are collapsed
to 5 exposure scores via ppo_to_exposure_scores before the backtest
loop. When CUDA is unavailable or the backtest fails, the adapter
falls back to the original -avg_episode_reward objective.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Flip --gpu-eval default to true (--no-gpu-eval to opt out)
- Move actions_history scatter-write into env kernel (zero CPU accumulation)
- Remove done-flag periodic download (env kernel handles per-thread)
- Only GPU→CPU transfer is final metrics readback (n_windows × 40 bytes)
- Add spread cost discrepancy warning when GPU path is active
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document known benign race on reset_flags (cross-block __threadfence
not being a grid-wide barrier) and writeback picking an arbitrary
episode's DSR/EMA as the representative seed. Remove dead reads for
epoch_port_value/pos/cash in both standard and warp-cooperative kernel
variants.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --gpu-eval flag that routes DQN fold evaluation through
GpuBacktestEvaluator instead of the per-bar CPU path. The GPU path
uploads the full test fold as a single window, runs the env kernel
loop on-device, and downloads only the final WindowMetrics (10 floats).
Key design choices:
- GPU path uses greedy argmax (not hierarchical softmax); results differ
slightly from CPU path — this is documented and intentional
- Falls back to CPU path on any GPU error (no silent failures)
- Also adds --initial-capital flag needed by GpuBacktestConfig
- Fix pre-existing clippy::wildcard_enum_match_arm in
GpuBacktestEvaluator::new (Device::_ → explicit variants)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds crates/ml/tests/gpu_backtest_validation.rs with 6 deterministic
tests that validate GpuBacktestEvaluator produces reasonable metrics:
always-long on uptrend (positive PnL), always-long on downtrend
(negative PnL), always-flat (~zero PnL), multi-window ordering,
extended metrics finiteness/self-consistency, and trade count.
All tests are #[ignore] gated and skip gracefully on CPU-only machines
(CI passes with 6 ignored; intended for GPU development machines).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements backtest_gather_kernel.cu (gather_states) which reads directly
from the pre-uploaded features buffer and live portfolio state on GPU,
eliminating the large GPU→CPU download of the full features buffer that
the old gather_states() path performed each step (n_windows×max_len×feat_dim
floats, e.g. 134 MB for 8 windows × 100k steps × 42 features).
The new path: kernel writes [n_windows, state_dim] into states_buf, then
only that tiny buffer (~1.5 KB for 8×48) is downloaded to create the
Candle tensor — a ~100,000x reduction in per-step data transfer.
Wiring changes in GpuBacktestEvaluator:
- Added GATHER_PTX OnceLock + compile_gather_ptx()
- Added gather_kernel (CudaFunction) and states_buf (CudaSlice<f32>) fields
- Added portfolio_dim field (always 3, validated in gather_states())
- Allocates states_buf = n_windows * (feature_dim + 3) in new()
- gather_states() now launches kernel then downloads small output buffer
- metrics download updated to 10 floats/window (Task 13 extended metrics)
- Added 3 new tests: gather PTX compilation, portfolio_dim validation,
state_dim calculation; all 10 gpu_backtest tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GPU-accelerated backtest path that runs walk-forward evaluation
entirely on GPU (env step kernel + metrics reduction), falling back
to the existing CPU path if CUDA is unavailable or the GPU path fails.
Changes:
- Make DQN::q_values_for_batch() public for external Q-value access
- Add RegimeConditionalDQN::batch_q_values() for regime-routed Q-values
- Add DQNAgentType::batch_q_values() dispatch method
- Add DQNTrainer::evaluate_gpu() method (#[cfg(feature = "cuda")])
- Wire GPU-first backtest at the decision point with CPU fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the GPU backtest evaluator (Task 9): uploads walk-forward
window data once, runs the step loop with Candle forward pass + env
kernel, then a single metrics reduction kernel with one final download.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compute_epoch_q_diagnostics now uses compute_q_diagnostics_gpu() free
function with Candle tensor ops. Batches gap stats + per-action means
into single 8-float readback instead of full N×5 to_vec2 download.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
One block per window. Parallel reduction for Sharpe, Sortino,
total PnL, max drawdown, win rate, trade count. Single kernel
launch reduces all windows simultaneously.
One thread per walk-forward window, parallel across all windows.
Handles: action→exposure mapping, trade execution with tx costs,
mark-to-market, step return calculation, drawdown tracking.
Portfolio state persists across steps in GPU global memory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DSR portfolio reset and normalizer reset now happen via kernel flags
instead of CPU state mutation. Eliminates cudaStreamSynchronize at
epoch boundaries.
Also fix brace mismatch in gpu_training_guard.rs that left the
accumulate_q_value/read_q_accumulator/reset_q_accumulator methods
outside the impl block (introduced by Task 4 in-progress work).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates N*8 bytes of memcpy_dtoh per experience kernel launch.
MonitoringReducer accumulates stats on GPU, single 48-byte download
at epoch boundary. Zero cudaStreamSynchronize during experience collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add n=0 early-return guard to reduce() to prevent CUDA divide-by-zero
- Add i32::MAX bounds check before casting n to prevent overflow
- Cache PTX compilation with OnceLock<Result<Ptx, String>> (compile once per process)
- Add Safety comment to unsafe block documenting slice/buffer invariants
- Fix doc comment: clarify 48-byte is GPU transfer size (12 × f32)
- Add shmem comment explaining kernel uses static __shared__ only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces per-launch rewards/actions download with single epoch-end
reduction. monitoring_reduce kernel computes mean, std, min, max,
Sharpe estimate, and per-action counts via parallel reduction.
Single 48-byte download instead of N*8 bytes per kernel launch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Episode-boundary resets previously used cold hardcoded constants (0.0f,
1e-8f) for DSR accumulators and EMA normalizer. Now warm-starts from
the persistent epoch state so all episodes within an epoch benefit from
accumulated statistics, not just the first.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The epoch_vol_ema, epoch_dsr_mean, and epoch_dsr_var variables were loaded
from global memory at kernel start but never wired into the actual computation.
The EMA normalizer (ema_mean/ema_var) and DSR accumulators (dsr_A/dsr_B) were
initialized with hardcoded constants, so epoch state round-tripped unchanged.
Changes:
- Seed ema_mean/ema_var from epoch_dsr_mean/epoch_dsr_var at kernel start
- Seed dsr_A/dsr_B from epoch_dsr_mean/epoch_dsr_var at kernel start
- Seed ema_init/dsr_initialized from epoch_step_count > 0 (skip cold start
on subsequent epochs)
- Add local_vol_ema/local_median_vol seeded from epoch_vol_ema/epoch_median_vol
- Update vol EMA each timestep from market feature index 3 (log close return),
mirroring CPU DQNTrainer::vol_ema / median_vol logic
- At writeback, write actual computed dsr_A/dsr_B or ema_mean/ema_var (conditioned
on use_dsr), and computed local_vol_ema/local_median_vol, instead of unmodified
loaded epoch values
- Applied identically to both dqn_full_experience_kernel (standard) and
dqn_full_experience_kernel_warp (warp-cooperative) variants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU PER replay buffer had a hardcoded 4 GB MAX_BYTES limit that
rejected the auto-sizer's 10M-entry proposal on H100 (80 GB VRAM).
Now per_max_buffer_bytes() computes 20% of total VRAM (min 1 GB) and
flows through OptimalReplayConfig → DQNConfig → GpuReplayBufferConfig
so both subsystems agree on the budget.
Also fixes misleading regime detection log (indices 211/219 → 40/41)
and renames dqn_config_2025 → dqn_default_config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminates cudaStreamSynchronize between epochs by keeping vol EMA,
portfolio state, and DSR normalizer in persistent CudaSlice buffers.
Kernel reads initial state at launch, writes final state at exit.
- Add epoch_state CudaSlice<f32>[8] field and reset_flags u32 bitfield
to GpuExperienceCollector struct
- Allocate epoch_state in new() with sensible defaults (vol_ema=0.01,
initial_capital for portfolio, dsr_var=1.0 to avoid div-by-zero)
- Pass epoch_state and reset_flags as final args to both kernel variants
(standard per-thread and warp-cooperative)
- Kernel: thread/lane 0 of block 0 applies reset flags atomically with
__threadfence, all threads read 8-float epoch state from L1-cached
global memory, last block writes back updated values at exit
- Auto-clear reset_flags after each launch (one-shot semantics)
- Add set_reset_flags(), clear_reset_flags(), epoch_state_gpu(),
rewards_gpu(), actions_gpu() public methods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In candle-core (git 671de1d), min(D) and max(D) return Result<Tensor>,
not Result<(Tensor, Tensor)>. Use flatten_all() then min(0)/max(0) for
scalar reduction instead of the tuple destructuring pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Task 14: evaluate_baseline is at crates/ml/examples/evaluate_baseline.rs
(not bin/fxt/src/commands/), fix cargo check command and git add path
- Task 15: process_bar_factored takes (usize, &OHLCVBarF32, &FactoredAction)
not (f64, usize, f64, f64), fix test to use correct types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
