Commit Graph

4 Commits

Author SHA1 Message Date
jgrusewski
6ba52425ea feat(infra): Argo workflow templates, drop cuDNN, GPU hotpath fixes
- Add compile-and-deploy, train-dqn/ppo/supervised WorkflowTemplates
- Add Argo Events (EventSource, Sensor, Service) for webhook triggers
- Add NetworkPolicy for compile-and-deploy pods (MinIO/DNS/API egress)
- Add convenience scripts: argo-compile-deploy.sh, argo-train.sh
- Drop cuDNN feature flags from all 9 ML crates (zero conv ops in codebase)
- Switch training runtime base to nvidia/cuda:12.9.1-runtime (saves ~800MB)
- Delete unused selective_scan.cu (16KB, zero Rust callers)
- Fix GPU hotpath violations in ml-core (NVTX, gradient utils, capabilities)
- Fix clippy warnings in ml-dqn (VarMap backticks, const fn)
- Add DQN GPU smoketest, backtest evaluator signal adapter fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 01:44:03 +01:00
jgrusewski
63d8619379 infra(docker): upgrade all images to Ubuntu 24.04 + CUDA 12.9
Align all 4 Docker images with local dev environment:
- ci-builder: CUDA 12.4.1/Ubuntu 22.04 → 12.9.1/Ubuntu 24.04
- ci-builder-cpu: Debian bookworm → Ubuntu 24.04 (+ explicit rustup)
- foxhunt-runtime: Debian bookworm → Ubuntu 24.04
- foxhunt-training-runtime: CUDA 12.6.3 → 12.9.1, nvrtc 12-6 → 12-9

All images now have glibc 2.39, matching the local build machine.
Binaries compiled locally or in CI will run in any of these containers
without GLIBC_2.3x version mismatch errors.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 09:34:47 +01:00
jgrusewski
7751f7615a fix(ci): unblock CPU service builds and fix H100 BF16 regime classification
Three fixes validated by 20/20 hyperopt trials on H100 (zero OOM):

1. Workspace default-features: ml-core, ml-dqn, ml-ppo, ml-supervised
   workspace deps now have default-features=false. Prevents cudarc
   (which requires nvcc) from leaking into CPU service builds via
   Cargo feature unification. CI compile-services was failing with
   "Failed to execute nvcc: No such file or directory" (exit 101).

2. BF16 comparison fix: Candle's gt()/le() don't support BF16 operands.
   Cast ADX/CUSUM features to F32 before threshold comparison in
   regime classification. Previous approach (cast threshold to BF16)
   failed due to Candle broadcast_as reverting dtype.

3. CI pipeline: expand ML change detection to all 14 sub-crates,
   add component:compile labels for sccache network policy matching,
   bump training runtime to CUDA 12.6 + Ubuntu 24.04 (glibc 2.39).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 18:11:04 +01:00
jgrusewski
7f3b511203 infra: add 3 generic base images for S3 binary share
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:57:19 +01:00