Argo artifact logs were stored in foxhunt-training-results which is
meant for model checkpoints. Separate bucket makes logs findable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test module used `super::*` but TradingState lives in
crate::dqn, not in the trainer module — broke `cargo test --lib`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove ~850 lines of unreachable CPU fallback code from 3 hot-path files:
- hyperopt/dqn.rs: delete 250-line CPU backtest path (GPU eval mandatory)
- evaluate_baseline.rs: delete CPU PPO eval + non-CUDA fallback (147 lines)
- trainers/ppo.rs: delete 6 dead CPU rollout methods (~370 lines),
split train() into dispatcher + #[cfg(feature = "cuda")] train_gpu()
All .to_vec1() GPU hot-path guard violations eliminated.
GPU failures are now hard errors, not silent CPU fallbacks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit tests need scalar readbacks for assertions — .to_scalar() is not
flagged but .to_vec1() in test modules would generate false positives.
Guard now correctly excludes inline #[cfg(test)] modules while checking
all production code including ensemble adapters.
Full --all scan reveals 31 pre-existing production violations across
ml-dqn, ml-ppo, ml-supervised, and ensemble adapters — separate cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtest_forward_kernel.cu calls q_forward_dueling_warp_shmem but
it was defined in dqn_experience_kernel.cu — a different compilation unit.
Move the function, TILE_LAYER_WARP_CLEAN macro, SHMEM_MIN and DIST_SIZE
to common_device_functions.cuh so both kernels can use them.
Add #ifndef guards to all macros to prevent redefinition warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI toolkit (CUDA 12.9, PTX ISA 8.7) requires:
1. cp.async.bulk needs .mbarrier::complete_tx::bytes completion mechanism
(mandatory since PTX ISA 8.3 / CUDA 12.3)
2. mbarrier.try_wait.parity.acquire needs .cta scope qualifier between
.acquire and .shared::cta
Reverts the DISABLE_TMA workaround — TMA now compiles natively to cubin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI CUDA toolkit's ptxas fails on TMA instructions (cp.async.bulk)
with "completion_mechanism modifier required". Prepend #define DISABLE_TMA 1
in compile_ptx_for_device() so ALL kernel compilations (experience collector,
backtest evaluator, PPO collector, curiosity trainer, action selector) use
the float4 cooperative load fallback instead of TMA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When commit-sha defaults to "HEAD", git checkout HEAD is a no-op — it keeps
the PVC's stale checkout. Now resolves HEAD to origin/{branch} after fetch,
ensuring the latest pushed code is always compiled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvcc now produces cubin (native SASS) with -cubin -arch=sm_XX instead of
PTX with -ptx -arch=compute_XX. cuModuleLoad() loads SASS directly — zero
driver JIT. TMA instructions (cp.async.bulk on sm_90+) work natively because
nvcc handles them during offline compilation. Removed TMA fallback from
experience collector (double-compile + 15s retry overhead eliminated).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
train_baseline_rl doesn't accept --initial-capital (only hyperopt and
evaluate binaries do). Removing it fixes the exit code 2 in DQN
hyperopt v3 train-best step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cast all Tensor::full() / Tensor::from_vec() call sites to training_dtype
instead of defaulting to F32. Fixes dtype mismatch errors (BF16 vs F32)
in PPO training on CUDA:
- tensor_ops: scalar_mul, clamp, normalize match operand dtype
- trajectories: TrajectoryBatch/MiniBatch to_tensors cast to training dtype
- continuous_ppo: ContinuousTrajectoryBatch/MiniBatch to_tensors cast
- adaptive_entropy: cast entropy to F32 for alpha multiplication boundary
- continuous_policy: forward() input cast, Tensor::full scalars match dtype
- flow_policy: sample_base_noise cast to training dtype
- hidden_state_manager: reset tensors use training_dtype
- ensemble/ppo adapter: predict input cast to training dtype
- trainable_adapter: test uses training_dtype instead of hardcoded F32
Verified: 198/198 ml-ppo tests pass, 63/63 ml PPO tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- backtest_forward_kernel: dim_overrides must precede common_device_functions.cuh
which has #error guards requiring STATE_DIM/MARKET_DIM/PORTFOLIO_DIM to be
defined before inclusion. Experience collector already had correct ordering.
- Cap MAX_EPISODES from 8192→4096 (diminishing returns above 4096, wastes walltime)
- Cap trainer .min() from 0x8000 (32768) → 4096 to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ci-builder image has CUDA stub libraries that shadow real NVIDIA
drivers. Without filtering stubs from LD_LIBRARY_PATH, training binaries
report "CUDA GPU required but none available" even on GPU nodes.
Add to hyperopt, train-best, and evaluate steps:
- LD_LIBRARY_PATH stubs filter (grep -v stubs)
- nvidia-smi verification
- RUST_LOG, SQLX_OFFLINE, CUBLAS_WORKSPACE_CONFIG env vars
- PATH includes /workspace/bin (use binary name, not full path)
Matches the existing pattern in training-workflow-template.yaml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix BF16/F32 dtype mismatch in PpoStrategy and PpoLstmStrategy
validation adapters. Tensor::from_vec(Vec<f32>) creates F32 tensors,
but PPO networks use BF16 weights on H100. Cast state_tensor to
training_dtype() at the boundary before passing to network forward.
Fixes 4 ppo-lib failures on H100:
- test_ppo_strategy_train_and_evaluate
- test_ppo_strategy_reset
- test_ppo_lstm_strategy_train_and_evaluate
- test_ppo_lstm_strategy_reset
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eventsource had no nodeSelector and landed on ci-compile-cpu
(POP2-HC-32C-64G, €0.56/hr), blocking autoscaler from scaling it
to 0. Now pinned to platform pool — ci-compile-cpu will autoscale
down, saving ~€410/mo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
20 epochs × 64 episodes × 200 timesteps = ~62 min per test on H100
with GPU experience collector enabled. Reduce to 10 epochs (~31 min)
to leave headroom for the remaining test suites within the 120-min
workflow deadline. Assertions remain equivalent (5% loss reduction,
Q-value divergence, checkpoint round-trip, walk-forward validation).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 concurrent DQN trainers on a shared H100 causes severe GPU contention
(28-39% utilization, 42 GB VRAM). Serializing with --test-threads=1
gives each test full GPU access, reducing total wallclock time despite
sequential execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The epsilon assertion expected <0.01 (epsilon=0.0 with noisy nets), but
the codebase evolved: noisy_epsilon_floor (0.05) now provides a minimum
exploration rate to prevent action collapse while NoisyNets handle the
primary learned exploration. Updated assertions to match: epsilon < 0.10.
Also reduced pipeline test epochs (10→5, 20→10) to prevent GPU timeout
when 5 concurrent DQN trainers share one H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reduce per-epoch GPU work from 4M to 12.8K experiences (64 episodes ×
200 timesteps) in CI integration tests. Still exercises the full fused
CUDA kernel (branching+C51+NoisyNets+DSR+fill-sim+N-step) but
completes within CI deadline. Production conservative() defaults
(8192×500) remain untouched.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DefaultHasher (SipHash) uses random per-process keys — the PTX cache
would never hit across separate cargo test runs. Switch to SHA-256
(deterministic) so cached PTX persists on the PVC across CI runs.
Also extend activeDeadlineSeconds from 90min to 120min to accommodate
the one-time cold-start NVRTC compilation (30+ min for the fused
experience collector kernel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fused DQN experience collector kernel (4490 lines: branching +
C51 + NoisyNets + fill sim + DSR + N-step) takes 30+ minutes to
compile via NVRTC on H100. This adds a PTX disk cache keyed by
SHA-256(arch, source) in $CARGO_TARGET_DIR/.ptx_cache/ (CI PVC).
Cold start pays the NVRTC cost once; all subsequent runs with
identical source + dimensions load cached PTX in <100ms.
Cache invalidates automatically when kernel source or network
dimensions change (different hash → cache miss → recompile).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fused NVRTC kernel (branching+C51+NoisyNets+DSR+fill-sim) takes
30+ min to compile at runtime on H100, causing CI tests to hit the
90-minute workflow deadline. Disable enable_gpu_experience_collector
in all integration tests that call DQNTrainer::train(). The GPU
experience collector is validated by lib tests (gpu_residency).
Training forward/backward/optimizer still runs on CUDA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integration tests (e2e DQN training on 148k bars) take 30+ minutes
in debug mode (opt-level=0). CARGO_PROFILE_TEST_OPT_LEVEL=1 enables
basic optimizations for test binaries — requires recompile but tests
run 5-10x faster. Local dev unaffected (env override only in CI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline validated end-to-end on H100: 3/3 steps green, 539 tests pass.
Switch podGC back to OnPodCompletion to auto-clean all pods after runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
7 tests call train_step() via CPU experience replay, but with the cuda
feature GPU PER is mandatory (no CPU path). Mark them
#[cfg_attr(feature = "cuda", ignore)] — the GPU training pipeline
integration tests cover this path properly on H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. CUDA_ERROR_STUB_LIBRARY: ci-builder image has CUDA stubs ahead of
real NVIDIA driver libs in LD_LIBRARY_PATH. Prepend
/usr/local/nvidia/lib64 (device-plugin mount) so real driver is
found first.
2. Git checkout: `git checkout --force main` stays on local main
without pulling. Add `git reset --hard origin/$REF` to fast-forward
branch to latest remote commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs in gpu-test-pipeline-template.yaml:
1. /bin/sh doesn't support bash-isms (<<<, read -ra, arrays) used in
the test runner script. Switch to /bin/bash.
2. --nocapture was appended as a cargo argument instead of a test binary
argument. Now detects whether -- separator exists in args and places
--nocapture correctly after it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: default-deny-all blocks egress from gpu-test pods to MinIO,
causing the Argo wait sidecar to hang on log upload. Fix: add egress
policy for gpu-test component (DNS, git, MinIO, Mattermost, registry).
Revert archiveLogs: false — logs should be stored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
