write-manifest needs compile-services and compile-training, but those
jobs use changes: rules. When a commit doesn't touch source code (e.g.
infra-only changes), those jobs don't exist in the pipeline, causing
GitLab to abort pipeline creation with 0 jobs for ALL stages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy job now triggers on infra/k8s/** and .gitlab-ci.yml changes,
not just source code changes. Ensures K8s manifest updates get applied.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Revert needs: [] back to needs: compile-training (optional). CI training
jobs should wait for fresh binaries when compile runs in the same
pipeline. Independent training via fxt CLI uses the gRPC path instead.
Keeps the writable bin dir fix (${CI_PROJECT_DIR}/bin/) and PATH export.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner pods run as non-root — can't write to /usr/local/bin/.
Download to ${CI_PROJECT_DIR}/bin/ and prepend to PATH instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs fetch binaries from S3 via rclone, not CI artifacts.
Remove needs: compile-training so training jobs can be triggered
independently without waiting for a compile stage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The rclone --include "*.{rl,supervised,baseline,uploader}*" glob does
not expand brace patterns like bash — silently downloads zero files.
Replace with unfiltered sync (bucket only has training binaries) and
wildcard chmod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
training-data-pvc is immutable (already bound with ReadWriteOnce),
and job-template.yaml has placeholder names (MODEL/TIMESTAMP) that
fail k8s validation. Only apply service deployments and binary cache
PVCs in the deploy job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job was using kubectl set image to stamp per-service Docker
images (api_gateway:<sha>, etc.) — these images no longer exist after
the S3 binary share migration. Replace with kubectl apply + rollout
restart, which picks up YAML changes and triggers initContainers to
fetch latest binaries from S3.
Delete redundant deploy-services job (merged into deploy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway quota limits POP2-32C-128G to 1 instance. Reduce each compile
job from 28→14 CPUs so compile-services and compile-training run in
parallel on the same 32-vCPU node instead of sequentially.
Also fix data-acquisition-service health probe: service doesn't implement
gRPC health protocol, switch to tcpSocket probe on port 50057.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web dashboard build now uses node:22-slim with inline rclone install
instead of a custom Kaniko-built image. This removes 1 Kaniko build
from the prepare stage, reducing pod scheduling contention.
Only 2 runtime images needed: CPU (foxhunt-runtime) and GPU
(foxhunt-training-runtime). Binary selection is via K8s command.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both images now use identical cuda:12.4.1-cudnn-runtime base, so maintaining
two separate Dockerfiles and build jobs was wasteful. Single image contains
all 7 training binaries, halving registry storage and Kaniko build time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
candle's CUDA backend links against libcudnn.so.9 for kernel operations.
Switch training-rl base from cuda:12.4.1-runtime to cuda:12.4.1-cudnn-runtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-training now needs compile-services. Both request 28 vCPUs but
the ci-compile-cpu pool (POP2-32C-128G) has only 32 vCPUs total.
Build-services jobs start as soon as compile-services finishes while
compile-training runs; no wall-clock penalty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-training was stuck on GPU node autoscale (L4 pool) but only
needs CUDA stubs for candle-kernels PTX generation, not a real GPU.
Switch to .rust-base-cpu (ci-compile-cpu pool) with ci-builder image
override for CUDA stubs. Unify pool labels — all compilation on CPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CI_BUILDER_CPU_IMAGE variable and build-ci-builder-cpu prepare job
- Add .rust-base-cpu template (no CUDA_COMPUTE_CAP)
- Split compile-services into compile-services (CPU) and compile-training (CUDA)
- Split .kaniko-base into .kaniko-service-base and .kaniko-training-base
- Service builds use build-out/services/, training uses build-out/training/
- Each compile job has conditional changes: rules for its source paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
infra-apply needs infra-plan, but infra-plan only runs when infra/**
changes. API/web-triggered pipelines fail to create because the
dependency doesn't exist. Adding optional: true allows infra-apply
to run without infra-plan for manual triggers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN/PPO networks are tiny (3 layers × 128 neurons). Running parallel
hyperopt on GPU wastes cores because CUDA context serializes across
threads — 5 trials on L4 only used 2000m of 6000m requested CPU.
Changes:
- Add --device flag to hyperopt_baseline_rl (auto/cpu/cuda)
- Auto mode forces CPU for parallel runs (no CUDA contention)
- CPU mode uses all available cores (no 2-core reserve)
- Add with_device() builder to DQN/PPO hyperopt trainers
- Downgrade "portfolio value <= 0" and GPU utilization warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule doesn't allow node-labels in kubelet_args. Instead, use the
auto-assigned k8s.scaleway.com/pool-name label that every node gets.
- Revert kubelet_args from Terraform (not supported)
- Switch runner node_selector: pool→k8s.scaleway.com/pool-name
- Update CI pipeline KUBERNETES_NODE_SELECTOR to match
- Helm upgraded both runners (CPU + GPU)
No more manual kubectl label after node scale-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL (DQN/PPO) stays on L4-1-24G — CPU-bound, low VRAM (<1GB).
Supervised stays on L40S-1-48G — GPU-bound, high VRAM.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All training jobs (RL + supervised) now use ci-training pool (L40S-1-48G)
instead of splitting RL→L4, supervised→L40S. With parallel hyperopt,
RL benefits from the L40S's more powerful GPU and same 8 vCPU.
Simplifies infrastructure: can potentially decommission L4 pool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL hyperopt runs parallel PSO trials across CPU cores. Previous 2000m
request underutilized the L4's 8 vCPUs. Bumped to 6000m/7800m so
hyperopt_baseline_rl --parallel 0 can saturate all cores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner defaults (16 CPU, 32Gi) are tuned for compile jobs but too high
for Kaniko image packaging. Only 2 build pods fit on the 32-core node,
queuing the remaining 7 with FailedScheduling: Insufficient cpu.
Set kaniko builds to 1000m/1Gi — all 9 can now run concurrently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Compile only needs CUDA stubs (baked into ci-builder) + CUDA_COMPUTE_CAP
for candle-kernels PTX generation. No nvidia-smi or real GPU needed.
- Hardcode CUDA_COMPUTE_CAP=89 (L4) in .rust-base variables
- Remove KUBERNETES_RUNTIME_CLASS_NAME from .rust-base
- compile-services: skip LD_LIBRARY_PATH stripping and nvidia-smi
- test: graceful fallback when nvidia-smi unavailable
This unblocks compile on nodes without nvidia RuntimeClass handler
and opens the door to using CPU-only nodes for faster compilation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner 18.9 expects KUBERNETES_NODE_SELECTOR_* values in
"key=value" format. Changed from bare values to "pool=services"
and "pool=ci-training".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner default nodeSelector changed from k8s.scaleway.com/pool-name to
simple pool=<name> label. CI variables now use correct format:
KUBERNETES_NODE_SELECTOR_pool: <value> (lowercase suffix = label key).
Fixes deploy job scheduling failure (wrong label format caused pods to
land on ci-compile instead of services pool).
Also fixes .train-rl-base tag from kapsule-rl (no matching runner) to
kapsule+gpu (matches existing runner).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy job was failing with "no runtime for nvidia" because the runner
set runtime_class_name=nvidia globally, but deploy routes to the
services pool which has no GPU/nvidia runtime.
Fix: removed global runtime_class_name from runner config, added
runtime_class_name_overwrite_allowed=".*", and set
KUBERNETES_RUNTIME_CLASS_NAME=nvidia in GPU job templates
(.rust-base, .train-rl-base, .train-validate-base).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable concurrent trial evaluation for DQN/PPO hyperparameter
optimization via clone-per-particle pattern — each PSO particle
clones the trainer and trains independently, replacing the previous
Arc<Mutex> serialization bottleneck. On L4 (8 vCPU) this yields
~4-5x throughput improvement.
Changes:
- DQNTrainer/PPOTrainer: Clone with Arc<AtomicUsize> trial counter
- DQNTrainer: replace unsafe mutable aliasing with Arc<Mutex> for
best_trial tracking
- ArgminOptimizer: add optimize_parallel() with ParallelObjectiveFunction
and scoped-thread LHS evaluation
- CLI: --parallel 0 (auto-detect CPUs-2), --initial-capital 35000,
--tx-cost-bps 0.1 (IBKR ES all-in)
- CI: both hyperopt jobs use --parallel 0 + IBKR ES cost defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL env simulation is single-threaded (~1 core actual). Lowering from
7000m to 2000m request allows DQN + PPO hyperopt to run simultaneously
on one L4 node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy only needs kubectl — no reason to consume GPU node CPU.
Routes to services pool with minimal 200m/500m CPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix ServiceAccount: use serviceAccount.name (not deprecated serviceAccountName)
- concurrent=1: one RL job gets full L4 node (7000m/7800m CPU, 16Gi/40Gi RAM)
- Faster iteration: ~2.3x more CPU per job vs splitting across 2 concurrent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add .train-rl-base template: no node selector (→ L4 default), CPU 3000m/3800m
- Update .train-validate-base: CPU 1000m/2000m for GPU-bound supervised models
- Switch train-validate-rl, hyperopt-ppo, hyperopt-dqn to .train-rl-base
- Reduce RL hyperopt epochs 15→8 for faster iteration (~2.4h vs 4.5h/trial)
- Supervised jobs unchanged: 8 models still on L40S with --epochs 15
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add changes: filters to compile-services and kaniko-base so pushes
that only touch infra/ (Terraform, Helm, K8s manifests) or
.gitlab-ci.yml don't trigger the 15-minute compile+build cycle.
Web/API pipeline triggers still run unconditionally for manual builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests are passing locally; skip them by default to unblock release cycles.
- test job: manual-only (was auto on push/MR)
- compile-services: always runs on main push (removed changes: filter)
- kaniko builds: require compile-services (was optional: true)
- kaniko builds: always run on main push (removed changes: filter)
This eliminates the L4 CPU scheduling conflict (test+compile both
requesting 7000m on a 7800m node) and gives a clean compile→build→deploy
pipeline for rapid iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set runtime_class_name=nvidia at runner level (all GPU pools have GPUs)
- Fix KUBERNETES_NODE_SELECTOR format: "key=value" not just value
- Replace wrong *_overwrite_allowed (regex) with correct
*_overwrite_max_allowed (max value) for CPU/memory overrides
- Remove per-job KUBERNETES_RUNTIME_CLASS_NAME (runner-level handles it)
- Simplify GPU detection in test/compile (GPU always present now)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
