The feature cache auto-invalidates via content hash — manual rm -f
wasted ~2 minutes of MBP-10 + OFI recomputation per trial start.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DSR warm-up: skip first 50 steps when EMA has insufficient history
- Phase Fast num_atoms: 51 → 101 (H100 can afford finer resolution,
1.19 per atom vs 2.35 — critical for distinguishing Q-values)
- Argo template: clear stale feature cache before hyperopt (ensures
fresh computation with VPIN/trades enrichment)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without CARGO_TARGET_DIR, cubin cache goes to /tmp (ephemeral).
With it, cubins persist at /workspace/.cubin_cache/ on the PVC
across runs — saves ~30s of nvcc compilation per trial.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_embedded_h100_parses: update assertions to match h100.toml values
(gpu_n_episodes=2048, gpu_timesteps_per_episode=100)
- dqn_training_smoke_test: apply dqn-smoketest profile to cap hidden_dim=32.
H100's gpu profile sets hidden_dim_base=256 which causes loss explosion
(375x in 3 epochs) with lr=0.001.
- Revert gpu-test-pipeline DAG to compile-and-test (RWO PVC constraint)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TOML training profiles are include_str!() into the ml binary at compile
time. Changes to config/training/ or config/gpu/ must trigger GPU test
rebuild, otherwise the binary runs with stale embedded defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Argo 'main' (emissary) container was the OOMKilled one, not 'wait'.
podSpecPatch now targets both containers to prevent executor OOM when
tracking large child workflow status JSON.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
submit-gpu-test wait sidecar had 64Mi limit — OOMKilled when tracking
large child workflow status JSON. Bumped to 256Mi.
Fixed YAML indentation errors in compile-and-train and training-pipeline
templates (misaligned labels, duplicate component keys).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 writes ${MODEL}_phase1_results.json to PVC, Phase 2 reads it
via --hyperopt-params. Phase 2 uses half the trials but double epochs.
train-best downstream reads final ${MODEL}_hyperopt_results.json unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 templates were missing app.kubernetes.io/component=gpu-test label,
causing MinIO log archival to fail (port 9000 blocked by network policy).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3Q of MBP-10 order book data for ES.FUT is ~18GB compressed.
10Gi PVC was insufficient for the full OHLCV + MBP-10 + trades dataset.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GPU test pipeline:
- Add perf-benchmark step after compile-and-test
- Runs DQN training on 3Q ES.FUT (OHLCV + MBP-10 + trades)
- Reports epoch time (ms), fails if > 500ms (H100 regression guard)
- batch_size=1024, 5 epochs × 100 steps, skips epoch 1 (init)
Populate test data job:
- Copy 3 quarters per symbol (was 1) for all data types
- OHLCV: ~6MB/symbol for walk-forward + perf benchmarks
- MBP-10: 3Q for OFI feature pipeline testing
- Trades: 3Q for trade-flow feature testing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DQN pipeline tests:
- Split dqn-pipeline into per-test cargo invocations in CI (CUDA Graph
capture corrupts async memory pool between sequential tests)
- Drop impl for GpuDqnTrainer: sync stream + destroy graph before buffers
- check_err drains in constructor and after graph capture
TFT fixes:
- forward_loss: reshape output [batch,horizon,quantiles] → [batch,quantiles]
to match target shape (fixes DimensionMismatch {expected:3, actual:3})
- smoke test: accept step=0 for models without backward support
- benchmark: remove hardcoded batch_size≤4 assertion (H100 can be larger)
Cleanup:
- Remove debug eprintln from elementwise.rs
- GPU-native cat bounds check
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts 4 commits (8139911c, 282f3aff, b3aca96d, dad61e4e) that
tried to workaround slow CI by limiting data subsets and cleaning
caches. The real issue is debug-mode .dbn.zst parsing taking minutes.
Kept: --test-threads=1 fix (root cause), hard CUDA error, action
range fixes, #[ignore] for heavy tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cargo incremental compilation on persistent PVC reuses old test
binaries even when source changed. Force cargo clean -p for ML crates
to ensure fresh compilation with latest max_bars and test fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: parallel test execution within a single cargo test binary
corrupts the CUDA primary context (cuDevicePrimaryCtxRetain race).
The dqn-smoke tests ran in parallel threads — 3 GPU tests using a
shared OnceLock<MlDevice> raced with smoke_e2e_dqn_training_loop
which creates a fresh CudaContext::new(0). The parallel context
init/teardown left the primary context in an error state, causing
subsequent cuInit(0) to fail silently.
Lib tests passed because they already had --test-threads=1.
Integration tests (dqn-smoke, dqn-smoke-train, ppo-barrier, etc.)
were missing it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
opt-level=1 caused DQN lib tests to timeout at the 2-hour deadline
(113 tests × debug-mode GPU init = too slow). opt-level=2 matches
local dev behavior.
Revert the cargo clean -p steps now that stale Candle cache is purged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Same stale incremental cache issue as ci-pipeline. Also saves compile
output to PVC for post-mortem debugging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pipe clippy and test output to /cargo-target/*.log via tee so errors
are readable after pod GC. Fixes blind CI failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Candle→cudarc migration invalidated all incremental compilation
artifacts for ml, ml-core, ml-dqn, ml-ppo, ml-supervised. CI PVC
retains stale .rmeta/.rlib causing spurious compile errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added scaleway_vpc_public_gateway + DHCP + gateway_network to TF
(was manually created, now codified with push_default_route=true)
- Added scripts/safe-node-replace.sh — one-at-a-time with DNS verification
- Added dns-bootstrap-policy.yaml — incident documentation + recovery procedure
- Bastion enabled (port 61000) for emergency SSH access
Prevention: NEVER replace all nodes at once. Use safe-node-replace.sh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ci-builder-cpu → ci-builder in test-gate template. cudarc 0.19 requires
nvcc at build time for its build.rs. Still runs on ci-compile-cpu node.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Argo artifact logs were stored in foxhunt-training-results which is
meant for model checkpoints. Separate bucket makes logs findable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When commit-sha defaults to "HEAD", git checkout HEAD is a no-op — it keeps
the PVC's stale checkout. Now resolves HEAD to origin/{branch} after fetch,
ensuring the latest pushed code is always compiled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
train_baseline_rl doesn't accept --initial-capital (only hyperopt and
evaluate binaries do). Removing it fixes the exit code 2 in DQN
hyperopt v3 train-best step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The ci-builder image has CUDA stub libraries that shadow real NVIDIA
drivers. Without filtering stubs from LD_LIBRARY_PATH, training binaries
report "CUDA GPU required but none available" even on GPU nodes.
Add to hyperopt, train-best, and evaluate steps:
- LD_LIBRARY_PATH stubs filter (grep -v stubs)
- nvidia-smi verification
- RUST_LOG, SQLX_OFFLINE, CUBLAS_WORKSPACE_CONFIG env vars
- PATH includes /workspace/bin (use binary name, not full path)
Matches the existing pattern in training-workflow-template.yaml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eventsource had no nodeSelector and landed on ci-compile-cpu
(POP2-HC-32C-64G, €0.56/hr), blocking autoscaler from scaling it
to 0. Now pinned to platform pool — ci-compile-cpu will autoscale
down, saving ~€410/mo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add deleteDelayDuration: 600s to podGC on CPU workflow templates and
sensors (ci-pipeline, compile-deploy, build-image). GPU training
workflows keep immediate cleanup to avoid wasting expensive GPU time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 concurrent DQN trainers on a shared H100 causes severe GPU contention
(28-39% utilization, 42 GB VRAM). Serializing with --test-threads=1
gives each test full GPU access, reducing total wallclock time despite
sequential execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DefaultHasher (SipHash) uses random per-process keys — the PTX cache
would never hit across separate cargo test runs. Switch to SHA-256
(deterministic) so cached PTX persists on the PVC across CI runs.
Also extend activeDeadlineSeconds from 90min to 120min to accommodate
the one-time cold-start NVRTC compilation (30+ min for the fused
experience collector kernel).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integration tests (e2e DQN training on 148k bars) take 30+ minutes
in debug mode (opt-level=0). CARGO_PROFILE_TEST_OPT_LEVEL=1 enables
basic optimizations for test binaries — requires recompile but tests
run 5-10x faster. Local dev unaffected (env override only in CI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline validated end-to-end on H100: 3/3 steps green, 539 tests pass.
Switch podGC back to OnPodCompletion to auto-clean all pods after runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. CUDA_ERROR_STUB_LIBRARY: ci-builder image has CUDA stubs ahead of
real NVIDIA driver libs in LD_LIBRARY_PATH. Prepend
/usr/local/nvidia/lib64 (device-plugin mount) so real driver is
found first.
2. Git checkout: `git checkout --force main` stays on local main
without pulling. Add `git reset --hard origin/$REF` to fast-forward
branch to latest remote commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs in gpu-test-pipeline-template.yaml:
1. /bin/sh doesn't support bash-isms (<<<, read -ra, arrays) used in
the test runner script. Switch to /bin/bash.
2. --nocapture was appended as a cargo argument instead of a test binary
argument. Now detects whether -- separator exists in args and places
--nocapture correctly after it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
