The ci-builder image has CUDA stub libraries that shadow real NVIDIA
drivers. Without filtering stubs from LD_LIBRARY_PATH, training binaries
report "CUDA GPU required but none available" even on GPU nodes.
Add to hyperopt, train-best, and evaluate steps:
- LD_LIBRARY_PATH stubs filter (grep -v stubs)
- nvidia-smi verification
- RUST_LOG, SQLX_OFFLINE, CUBLAS_WORKSPACE_CONFIG env vars
- PATH includes /workspace/bin (use binary name, not full path)
Matches the existing pattern in training-workflow-template.yaml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix BF16/F32 dtype mismatch in PpoStrategy and PpoLstmStrategy
validation adapters. Tensor::from_vec(Vec<f32>) creates F32 tensors,
but PPO networks use BF16 weights on H100. Cast state_tensor to
training_dtype() at the boundary before passing to network forward.
Fixes 4 ppo-lib failures on H100:
- test_ppo_strategy_train_and_evaluate
- test_ppo_strategy_reset
- test_ppo_lstm_strategy_train_and_evaluate
- test_ppo_lstm_strategy_reset
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cast input to weight dtype in DQN residual, rmsnorm, noisy_layers
- Set use_gpu=true in QNetworkConfig defaults and all config sites
- Resolve BF16 boundary mismatches in attention, curiosity, branching,
distributional_dueling across ml-dqn
- GPU-resident regime ops with BF16 boundary casts, eliminate .expect() in CUDA paths
- Eliminate all Device::Cpu fallbacks — GPU-only across 10 ML crates
- PPO: cast logits to F32 before softmax, cast batch tensors to training dtype
- Gradient collapse detection for RegimeConditionalDQN
- Wire halt_grad_collapse from CUDA guard kernel to halt training
- Dead neuron detection uses active network VarMap + squeeze factored readback
- Increment gradient_logging_step in GPU PER path
- Gradient collapse warmup guards use original buffer_size
- Cap training steps per epoch + tracing migration
- Replace Tensor::all() with sum_all() for pinned Candle compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eventsource had no nodeSelector and landed on ci-compile-cpu
(POP2-HC-32C-64G, €0.56/hr), blocking autoscaler from scaling it
to 0. Now pinned to platform pool — ci-compile-cpu will autoscale
down, saving ~€410/mo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
20 epochs × 64 episodes × 200 timesteps = ~62 min per test on H100
with GPU experience collector enabled. Reduce to 10 epochs (~31 min)
to leave headroom for the remaining test suites within the 120-min
workflow deadline. Assertions remain equivalent (5% loss reduction,
Q-value divergence, checkpoint round-trip, walk-forward validation).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 concurrent DQN trainers on a shared H100 causes severe GPU contention
(28-39% utilization, 42 GB VRAM). Serializing with --test-threads=1
gives each test full GPU access, reducing total wallclock time despite
sequential execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The epsilon assertion expected <0.01 (epsilon=0.0 with noisy nets), but
the codebase evolved: noisy_epsilon_floor (0.05) now provides a minimum
exploration rate to prevent action collapse while NoisyNets handle the
primary learned exploration. Updated assertions to match: epsilon < 0.10.
Also reduced pipeline test epochs (10→5, 20→10) to prevent GPU timeout
when 5 concurrent DQN trainers share one H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reduce per-epoch GPU work from 4M to 12.8K experiences (64 episodes ×
200 timesteps) in CI integration tests. Still exercises the full fused
CUDA kernel (branching+C51+NoisyNets+DSR+fill-sim+N-step) but
completes within CI deadline. Production conservative() defaults
(8192×500) remain untouched.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DefaultHasher (SipHash) uses random per-process keys — the PTX cache
would never hit across separate cargo test runs. Switch to SHA-256
(deterministic) so cached PTX persists on the PVC across CI runs.
Also extend activeDeadlineSeconds from 90min to 120min to accommodate
the one-time cold-start NVRTC compilation (30+ min for the fused
experience collector kernel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fused DQN experience collector kernel (4490 lines: branching +
C51 + NoisyNets + fill sim + DSR + N-step) takes 30+ minutes to
compile via NVRTC on H100. This adds a PTX disk cache keyed by
SHA-256(arch, source) in $CARGO_TARGET_DIR/.ptx_cache/ (CI PVC).
Cold start pays the NVRTC cost once; all subsequent runs with
identical source + dimensions load cached PTX in <100ms.
Cache invalidates automatically when kernel source or network
dimensions change (different hash → cache miss → recompile).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fused NVRTC kernel (branching+C51+NoisyNets+DSR+fill-sim) takes
30+ min to compile at runtime on H100, causing CI tests to hit the
90-minute workflow deadline. Disable enable_gpu_experience_collector
in all integration tests that call DQNTrainer::train(). The GPU
experience collector is validated by lib tests (gpu_residency).
Training forward/backward/optimizer still runs on CUDA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integration tests (e2e DQN training on 148k bars) take 30+ minutes
in debug mode (opt-level=0). CARGO_PROFILE_TEST_OPT_LEVEL=1 enables
basic optimizations for test binaries — requires recompile but tests
run 5-10x faster. Local dev unaffected (env override only in CI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline validated end-to-end on H100: 3/3 steps green, 539 tests pass.
Switch podGC back to OnPodCompletion to auto-clean all pods after runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
7 tests call train_step() via CPU experience replay, but with the cuda
feature GPU PER is mandatory (no CPU path). Mark them
#[cfg_attr(feature = "cuda", ignore)] — the GPU training pipeline
integration tests cover this path properly on H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. CUDA_ERROR_STUB_LIBRARY: ci-builder image has CUDA stubs ahead of
real NVIDIA driver libs in LD_LIBRARY_PATH. Prepend
/usr/local/nvidia/lib64 (device-plugin mount) so real driver is
found first.
2. Git checkout: `git checkout --force main` stays on local main
without pulling. Add `git reset --hard origin/$REF` to fast-forward
branch to latest remote commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs in gpu-test-pipeline-template.yaml:
1. /bin/sh doesn't support bash-isms (<<<, read -ra, arrays) used in
the test runner script. Switch to /bin/bash.
2. --nocapture was appended as a cargo argument instead of a test binary
argument. Now detects whether -- separator exists in args and places
--nocapture correctly after it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: default-deny-all blocks egress from gpu-test pods to MinIO,
causing the Argo wait sidecar to hang on log upload. Fix: add egress
policy for gpu-test component (DNS, git, MinIO, Mattermost, registry).
Revert archiveLogs: false — logs should be stored.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The wait sidecar blocks indefinitely trying to save logs to MinIO which
has no running pods. Disable log archiving on all 3 templates since test
output is captured in Argo's pod logs already.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates step failed because the service account lacked
permissions to manage configmaps (auto-compile-configmap.yaml) and
networkpolicies (argo-workflow-netpol.yaml) in the foxhunt namespace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
busybox doesn't support Argo's emissary executor protocol, causing the
wait sidecar to hang indefinitely and hold the GPU allocation. Switch to
foxhunt-runtime which has proper /bin/sh and lets the sidecar detect
container exit. Also adds nvidia-smi check during warmup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove #[ignore] from 3 Redis tests and use REDIS_URL env var from CI.
Add Redis 7 sidecar to test-gate pod with readiness probe + nc wait loop.
Tests gracefully skip if Redis unavailable (local dev without Docker).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- B1: Add LIB_FILTER mapping for tggn→tgnn module name (prevents zero-test false positive)
- W1: Replace ml_supervised:: imports with canonical ml:: paths in supervised_gpu_smoke_test
- W4: Use clean pass/fail exit code instead of raw failure count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test-gate script runs under /bin/sh (dash), which doesn't support
${PIPESTATUS[0]}. This caused 'Bad substitution' (exit 2) on EVERY CI
run regardless of test results.
Fix: redirect cargo test to file, capture $?, then cat for log output.
This is POSIX-compatible and correctly captures the test exit code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus registry.register() returns AlreadyReg when another test
thread triggers the lazy_static counters first. Both Ok and AlreadyReg
are valid — only hard errors indicate a real problem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Task 3b: GPU smoke test for all 8 supervised models via UnifiedTrainable
- Wire supervised_gpu_smoke_test into Task 4 workflow shell script (replaces wildcard)
- Each supervised model gets explicit test filter: test_${MODEL}_gpu_smoke
- Retain fallback for model-specific _integration tests if they exist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three Redis tests were running unconditionally on CI despite requiring
a local Redis server. They spin on connection attempts causing 60s+
timeouts and eventual test-gate failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
