Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1
Jobs, collect results via a Rust sidecar uploader, and support model
promotion with operator approval via fxt CLI.
- K8s dispatcher creates Jobs on gpu-training pool with native sidecar
- training_uploader crate: watches DONE/FAILED marker, uploads to S3,
reports completion via ReportJobCompletion gRPC
- PromotionManager compares metrics, queues better models for approval
- 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion
- fxt commands: train start, model list/approve/reject
- Training binaries write DONE/FAILED markers + metrics.json
- Dockerfile, K8s job template, and CI pipeline updated
- StartTraining gracefully falls back to in-process when outside K8s
- 27 new tests (16 service + 11 promotion), 141 total service tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was silently failing — bitnami/kubectl image lacks scw CLI
needed by Kapsule kubeconfig credential plugin. Now uses infra-runner
(has scw), installs kubectl inline until image is rebuilt. Adds
cluster-info check before deploy and proper error reporting per service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training image was recompiling from scratch inside Kaniko (~16+ min)
without sccache. Now training binaries are built in compile-services
with --features ml/cuda and PVC sccache (warm cache from service
binaries), then packaged into a CUDA runtime image (~30s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --cache=true --cache-repo to all 12 Kaniko builds
- Cache Docker layers in Scaleway CR (rg.fr-par.scw.cloud/foxhunt-ci/cache)
- Add Docker Hub auth to devcontainer + infra-runner prepare jobs
- First build populates cache; subsequent builds skip base image pulls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache 0.8+ (opendal backend) requires AWS_REGION, not just
AWS_DEFAULT_REGION. Also set SCCACHE_REGION as belt-and-suspenders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update COPY paths in all 3 Dockerfiles for crates/bin/services/testing layout
- Migrate service image builds from internal GitLab registry to SCW CR
- Update Kaniko auth to use SCW credentials (nologin + SCW_SECRET_KEY)
- Remove --insecure-registry flags (SCW CR is HTTPS)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate links libcuda.so.1 at runtime (via candle CUDA).
GPU-less CI nodes need the stub library so binaries can load, but
model_loader tests that actually use CUDA must be skipped. These will
run on gpu-training nodes separately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate dynamically links to libcuda.so.1 via candle.
NVIDIA CUDA dev images include stubs at /usr/local/cuda/lib64/stubs/
but only as libcuda.so (not .so.1). Create the symlink both in the
Dockerfile (for future builds) and inline in the test script (for
the current image).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
--run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
(pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container
All 10 pods (3 databases + 7 services) now 1/1 Running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dockerfile.service: unified multi-stage with sccache S3 support
- Dockerfile.web-gateway: Node dashboard + Rust gateway combined
- Dockerfile.training: CUDA 12.4 with H100 target (compute cap 9.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
