CI PVC structure is /data/test-data/ohlcv/ES.FUT, local is
test_data/ES.FUT. test_data_dir() now searches both layouts
and checks both FOXHUNT_TEST_DATA and TEST_DATA_DIR env vars.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause 1 — CUDA_ERROR_ILLEGAL_ADDRESS:
shmem_max_in_dim only included trunk dims (state_dim, shared_h1,
shared_h2) but not head dims (value_h, adv_h). BF16 weight tile
for branch output overflowed shared memory on RTX 3050 (48KB).
Root cause 2 — CUDA_ERROR_INVALID_VALUE on EMA kernel:
cudarc 0.17's automatic event tracking records read/write events on
CudaSlice buffers. During CUDA Graph capture (events disabled) then
replay (events re-enabled), stale write events from CudaSlice Drops
poison the context error_state. Next bind_to_thread() propagates it.
Fix: disable_event_tracking() at GpuDqnTrainer construction —
single-owner forked stream, all sync points are explicit.
Also:
- Remove all #[ignore] from smoke tests, use real ES.FUT .dbn data
- Validate DBN schema at file level (skip non-OHLCV)
- Organize test_data/ into per-symbol subdirectories
- Fix pre-existing gpu_kernel_parity_test + evaluate_baseline errors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MBP-10 .dbn files mixed into test_data/ caused OHLCV loader to crash
on unknown RType 0x42. Now silently skips non-OHLCV records.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.
Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After removing ensure_training_dtype(), forward methods that receive F32
inputs from tests/callers now fail with dtype mismatch against BF16 weights.
Add to_dtype(BF16) at forward entry of xLSTM (slstm, mlstm, block, network),
CfC cell, and diffusion time embedding. Fix quantization to accept BF16
tensors by casting to F32 before INT8 conversion. Update guard to catch
#[cfg(not(feature = "cuda"))] dead code. Bump DQN emergency_safe_defaults
replay_buffer_capacity from 1000 to 2048 (GPU PER floor = 1024).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete every CPU fallback block across 14 files (-286 lines):
- dqn.rs: CPU replay buffer, tensor construction, PER fallback, gradient paths
- hyperopt/adapters/ppo.rs: CPU curiosity modules, trajectory generation
- hyperopt/adapters/dqn.rs: CPU backtest fallback, sync no-op
- trainers/ppo.rs: CPU training error stub
- l2_cache.rs: CPU stub functions (gate callers behind cuda too)
- replay_buffer_type.rs: CPU is_gpu_prioritized fallback
- validation/harness.rs: CPU regime breakdown fallback
- build.rs: cpu_only_build cfg (never referenced)
- evaluate_baseline.rs: CPU gpu_handled fallback
- testing/integration/gpu: CPU test stubs
Zero #[cfg(not(feature = "cuda"))] remains in the codebase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AutoReplaySizer inflated smoke test buffer_size=500 to 10M on H100 (75GB VRAM),
causing GPU PER empty-sample failures and multi-hour hangs. Root cause: the sizer's
100K minimum was always above the test buffer size, so the guard never triggered.
Fixes:
- constructor.rs: skip auto-sizing when buffer_size < 100K (sizer's own minimum)
- config.rs: insert_batch_tensors falls back to CPU PER (download + add_batch)
when GPU PER unavailable, instead of hard error
- train_step.rs: skip fused CUDA Graph init when GPU PER not active (stream
capture can't include CPU→GPU transfers from CPU PER sampling)
- dqn.rs: allow CPU tensor construction path on CUDA when GPU PER falls back;
remove hard errors in PER priority update (2 locations)
- metrics.rs: fix portfolio tensor shape (broadcast_left→expand for 2D concat);
add CPU PER fallback for Q-value statistics sampling
- agent.rs, entropy_regularization.rs: GPU-native Gumbel-max via Tensor::rand
(eliminates per-call CPU Vec allocation + GPU upload)
- smoke test helpers: defense-in-depth replay_buffer_vram_fraction = 0.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvcc (C++11 mode) parses `25f` as an attempt to use user-defined literal
suffix `f` instead of the standard float suffix. When V_MIN/V_MAX are
injected as integer-formatted values (e.g., `-25f` from Rust's Display
trait on f32), nvcc fails with "user-defined literal operator not found".
Fix: use `{v_min:.1}f` format to guarantee a decimal point (`-25.0f`),
which is unambiguously a float literal in all C++ standards.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Non-DQN kernels (curiosity, PPO, epsilon-greedy, backtest-PPO) include
common_device_functions.cuh but don't define SHARED_H1/SHARED_H2/VALUE_H/ADV_H.
The unguarded q_forward_dueling_warp_shmem() references these constants,
causing nvcc compilation failures (undefined identifiers + cascading
"user-defined literal operator not found" errors).
Wraps the function in #if defined(SHARED_H1) && defined(SHARED_H2) &&
defined(VALUE_H) && defined(ADV_H) ... #endif so it's only compiled
when the DQN architecture constants are injected.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The BF16 matvec helpers (warp_matvec_bf16_shmem, warp_matvec_bf16_broadcast_shmem)
call warp_reduce_sum_all which is defined later in the header. nvcc on H100
rejects this without a forward declaration — broke experience kernel compilation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo artifact logs were stored in foxhunt-training-results which is
meant for model checkpoints. Separate bucket makes logs findable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test module used `super::*` but TradingState lives in
crate::dqn, not in the trainer module — broke `cargo test --lib`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove ~850 lines of unreachable CPU fallback code from 3 hot-path files:
- hyperopt/dqn.rs: delete 250-line CPU backtest path (GPU eval mandatory)
- evaluate_baseline.rs: delete CPU PPO eval + non-CUDA fallback (147 lines)
- trainers/ppo.rs: delete 6 dead CPU rollout methods (~370 lines),
split train() into dispatcher + #[cfg(feature = "cuda")] train_gpu()
All .to_vec1() GPU hot-path guard violations eliminated.
GPU failures are now hard errors, not silent CPU fallbacks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit tests need scalar readbacks for assertions — .to_scalar() is not
flagged but .to_vec1() in test modules would generate false positives.
Guard now correctly excludes inline #[cfg(test)] modules while checking
all production code including ensemble adapters.
Full --all scan reveals 31 pre-existing production violations across
ml-dqn, ml-ppo, ml-supervised, and ensemble adapters — separate cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtest_forward_kernel.cu calls q_forward_dueling_warp_shmem but
it was defined in dqn_experience_kernel.cu — a different compilation unit.
Move the function, TILE_LAYER_WARP_CLEAN macro, SHMEM_MIN and DIST_SIZE
to common_device_functions.cuh so both kernels can use them.
Add #ifndef guards to all macros to prevent redefinition warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI toolkit (CUDA 12.9, PTX ISA 8.7) requires:
1. cp.async.bulk needs .mbarrier::complete_tx::bytes completion mechanism
(mandatory since PTX ISA 8.3 / CUDA 12.3)
2. mbarrier.try_wait.parity.acquire needs .cta scope qualifier between
.acquire and .shared::cta
Reverts the DISABLE_TMA workaround — TMA now compiles natively to cubin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI CUDA toolkit's ptxas fails on TMA instructions (cp.async.bulk)
with "completion_mechanism modifier required". Prepend #define DISABLE_TMA 1
in compile_ptx_for_device() so ALL kernel compilations (experience collector,
backtest evaluator, PPO collector, curiosity trainer, action selector) use
the float4 cooperative load fallback instead of TMA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When commit-sha defaults to "HEAD", git checkout HEAD is a no-op — it keeps
the PVC's stale checkout. Now resolves HEAD to origin/{branch} after fetch,
ensuring the latest pushed code is always compiled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvcc now produces cubin (native SASS) with -cubin -arch=sm_XX instead of
PTX with -ptx -arch=compute_XX. cuModuleLoad() loads SASS directly — zero
driver JIT. TMA instructions (cp.async.bulk on sm_90+) work natively because
nvcc handles them during offline compilation. Removed TMA fallback from
experience collector (double-compile + 15s retry overhead eliminated).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
train_baseline_rl doesn't accept --initial-capital (only hyperopt and
evaluate binaries do). Removing it fixes the exit code 2 in DQN
hyperopt v3 train-best step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cast all Tensor::full() / Tensor::from_vec() call sites to training_dtype
instead of defaulting to F32. Fixes dtype mismatch errors (BF16 vs F32)
in PPO training on CUDA:
- tensor_ops: scalar_mul, clamp, normalize match operand dtype
- trajectories: TrajectoryBatch/MiniBatch to_tensors cast to training dtype
- continuous_ppo: ContinuousTrajectoryBatch/MiniBatch to_tensors cast
- adaptive_entropy: cast entropy to F32 for alpha multiplication boundary
- continuous_policy: forward() input cast, Tensor::full scalars match dtype
- flow_policy: sample_base_noise cast to training dtype
- hidden_state_manager: reset tensors use training_dtype
- ensemble/ppo adapter: predict input cast to training dtype
- trainable_adapter: test uses training_dtype instead of hardcoded F32
Verified: 198/198 ml-ppo tests pass, 63/63 ml PPO tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- backtest_forward_kernel: dim_overrides must precede common_device_functions.cuh
which has #error guards requiring STATE_DIM/MARKET_DIM/PORTFOLIO_DIM to be
defined before inclusion. Experience collector already had correct ordering.
- Cap MAX_EPISODES from 8192→4096 (diminishing returns above 4096, wastes walltime)
- Cap trainer .min() from 0x8000 (32768) → 4096 to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ci-builder image has CUDA stub libraries that shadow real NVIDIA
drivers. Without filtering stubs from LD_LIBRARY_PATH, training binaries
report "CUDA GPU required but none available" even on GPU nodes.
Add to hyperopt, train-best, and evaluate steps:
- LD_LIBRARY_PATH stubs filter (grep -v stubs)
- nvidia-smi verification
- RUST_LOG, SQLX_OFFLINE, CUBLAS_WORKSPACE_CONFIG env vars
- PATH includes /workspace/bin (use binary name, not full path)
Matches the existing pattern in training-workflow-template.yaml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix BF16/F32 dtype mismatch in PpoStrategy and PpoLstmStrategy
validation adapters. Tensor::from_vec(Vec<f32>) creates F32 tensors,
but PPO networks use BF16 weights on H100. Cast state_tensor to
training_dtype() at the boundary before passing to network forward.
Fixes 4 ppo-lib failures on H100:
- test_ppo_strategy_train_and_evaluate
- test_ppo_strategy_reset
- test_ppo_lstm_strategy_train_and_evaluate
- test_ppo_lstm_strategy_reset
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eventsource had no nodeSelector and landed on ci-compile-cpu
(POP2-HC-32C-64G, €0.56/hr), blocking autoscaler from scaling it
to 0. Now pinned to platform pool — ci-compile-cpu will autoscale
down, saving ~€410/mo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
