- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Source: docs.nvidia.com/nsight-systems/InstallationGuide
Repo: developer.download.nvidia.com/devtools/repos/ubuntu2404/amd64/
Key: 7fa2af80.pub from CUDA repo
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enables nsys profiling for CUDA graph node-level traces on H100.
Use --sanitizer nsys in argo-train.sh to activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: weight tensors packed sequentially in the flat params buffer
had non-aligned start offsets when preceding tensors had odd element counts
(e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with
CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers.
Fix: pad each tensor to 4-element boundary (16 bytes) in both
f32_weight_ptrs_from_base (pointer computation) and compute_total_params
(buffer allocation). Added align4() and padded_byte_offset() helpers,
fixed shrink_perturb skip range and bottleneck gradient offset.
Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
(forward + backward). Explicit TF32 tensor core path, required by cuBLAS
13.0 on H100 SM90.
Deleted dead bf16_weight_ptrs function.
19/19 smoke tests pass on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cuda-nvcc-12-4 + cuda-cudart-dev-12-4 (~400MB) added to the CPU builder.
Reverts compile-and-deploy back to ci-builder-cpu (was briefly switched
to full ci-builder which is 6GB+ and unnecessary).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Wire build-image endpoint into EventSource + Sensor for manual triggers
- Add rebuild-* DAG tasks to compile-and-deploy and compile-and-train
templates — Kaniko layer cache makes cached builds ~15-30s
- Fix foxhunt-runtime Dockerfile: rename ubuntu user instead of groupadd
(GID 1000 already exists in ubuntu:24.04)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No cudnn feature flags remain after 6ba52425 — the devel headers
and libs were dead weight (~1.5 GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply-argo-templates needs git (clone repo) + kubectl (apply manifests).
ci-builder-cpu has git but lacked kubectl — added it. foxhunt-runtime
has kubectl but no git and runs as non-root, so can't be used here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ubuntu 24.04 minimal doesn't include passwd (useradd/groupadd).
Debian bookworm-slim had it by default. Required for foxhunt user.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9
All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):
1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
workspace deps now have default-features=false. Prevents cudarc
(which requires nvcc) from leaking into CPU service builds via
Cargo feature unification. CI compile-services was failing with
"Failed to execute nvcc: No such file or directory" (exit 101).
2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
Cast ADX/CUSUM features to F32 before threshold comparison in
regime classification. Previous approach (cast threshold to BF16)
failed due to Candle broadcast_as reverting dtype.
3. CI pipeline: expand ML change detection to all 14 sub-crates,
add component:compile labels for sccache network policy matching,
bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kubectl v1.31.4 to foxhunt-runtime Dockerfile so deploy step
doesn't need to download it each run. Deploy step falls back to
curl download if kubectl not found (for current image version).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
detect-changes was falling back to Cargo.toml (rebuild-all) because jq
is not installed in foxhunt-runtime:latest. Replaced with grep+sed
extraction that needs no external dependencies. Also added jq to the
Dockerfile for future use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web dashboard build now uses node:22-slim with inline rclone install
instead of a custom Kaniko-built image. This removes 1 Kaniko build
from the prepare stage, reducing pod scheduling contention.
Only 2 runtime images needed: CPU (foxhunt-runtime) and GPU
(foxhunt-training-runtime). Binary selection is via K8s command.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collection kernels (dqn_experience_kernel.cu,
ppo_experience_kernel.cu) use cudarc::nvrtc::compile_ptx() at runtime.
Without libnvrtc.so the kernel compile fails silently and falls back to
CPU experience collection. Adding cuda-nvrtc-12-4 (~30MB) enables full
GPU-accelerated experience collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both images now use identical cuda:12.4.1-cudnn-runtime base, so maintaining
two separate Dockerfiles and build jobs was wasteful. Single image contains
all 7 training binaries, halving registry storage and Kaniko build time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
candle's CUDA backend links against libcudnn.so.9 for kernel operations.
Switch training-rl base from cuda:12.4.1-runtime to cuda:12.4.1-cudnn-runtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-services stage outputs binaries to build-out/services/ but
the web-gateway Dockerfile still referenced the old build-out/ path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Based on rust:1.89-slim-bookworm (~2-3GB vs ~8GB CUDA devel).
Same toolchain: mold 2.35, protoc 28.3, sccache 0.10, clang, lld.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New evaluate_supervised binary runs walk-forward inference on supervised
model checkpoints (TFT, Mamba2, etc.), converts directional predictions
to trading signals, and computes Sharpe/MaxDD/WinRate/DirAccuracy.
CI changes:
- train-validate-dqn → train-validate-rl (trains+evals DQN+PPO)
- train-validate-tft now runs evaluate_supervised after training
- web/api fallback rules added to train-validate and deploy stages
- evaluate_supervised added to compile-services and Docker images
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1
Jobs, collect results via a Rust sidecar uploader, and support model
promotion with operator approval via fxt CLI.
- K8s dispatcher creates Jobs on gpu-training pool with native sidecar
- training_uploader crate: watches DONE/FAILED marker, uploads to S3,
reports completion via ReportJobCompletion gRPC
- PromotionManager compares metrics, queues better models for approval
- 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion
- fxt commands: train start, model list/approve/reject
- Training binaries write DONE/FAILED markers + metrics.json
- Dockerfile, K8s job template, and CI pipeline updated
- StartTraining gracefully falls back to in-process when outside K8s
- 27 new tests (16 service + 11 promotion), 141 total service tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was silently failing — bitnami/kubectl image lacks scw CLI
needed by Kapsule kubeconfig credential plugin. Now uses infra-runner
(has scw), installs kubectl inline until image is rebuilt. Adds
cluster-info check before deploy and proper error reporting per service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training image was recompiling from scratch inside Kaniko (~16+ min)
without sccache. Now training binaries are built in compile-services
with --features ml/cuda and PVC sccache (warm cache from service
binaries), then packaged into a CUDA runtime image (~30s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --cache=true --cache-repo to all 12 Kaniko builds
- Cache Docker layers in Scaleway CR (rg.fr-par.scw.cloud/foxhunt-ci/cache)
- Add Docker Hub auth to devcontainer + infra-runner prepare jobs
- First build populates cache; subsequent builds skip base image pulls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache 0.8+ (opendal backend) requires AWS_REGION, not just
AWS_DEFAULT_REGION. Also set SCCACHE_REGION as belt-and-suspenders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update COPY paths in all 3 Dockerfiles for crates/bin/services/testing layout
- Migrate service image builds from internal GitLab registry to SCW CR
- Update Kaniko auth to use SCW credentials (nologin + SCW_SECRET_KEY)
- Remove --insecure-registry flags (SCW CR is HTTPS)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate links libcuda.so.1 at runtime (via candle CUDA).
GPU-less CI nodes need the stub library so binaries can load, but
model_loader tests that actually use CUDA must be skipped. These will
run on gpu-training nodes separately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate dynamically links to libcuda.so.1 via candle.
NVIDIA CUDA dev images include stubs at /usr/local/cuda/lib64/stubs/
but only as libcuda.so (not .so.1). Create the symlink both in the
Dockerfile (for future builds) and inline in the test script (for
the current image).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
--run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
