Two 9-quarter precompute_features runs OOM-killed on the existing
ci-compile-cpu pool (POP2-HC-32C-64G, 56Gi cgroup limit):
- train-wq8b8: exit 137 ~12s after "OFI computed" at ~57Gi
- train-2l6p4: exit 137 ~12s after "OFI computed" at ~52Gi peak
(despite the into_iter + drop(feature_vectors) +
normalize_in_place refactors landed in a27cb40a9 + 623ebfcf7,
which trimmed ~12GB of avoidable retention)
The remaining ~52-60GB peak is the irreducible working set at the
post-OFI / pre-alpha-pipeline step:
- 9-quarter MBP-10 snapshots (~10GB)
- 199M front-month trades as Mbp10Trade (~13GB)
- 17.8M bars × features + targets + OFI buffers (~14GB)
- alpha_snapshots (move-handoff from all_snapshots, ~10GB)
- feature/target Vecs (~7GB) + Rust allocator overhead
Adds a dedicated high-memory pool sized for this rare path:
- New scaleway_k8s_pool.ci_compile_cpu_hm (POP2-HM-32C-256G,
32 vCPU + 256GB RAM) with size=0 + min_size=0 autoscaling. Costs
zero when idle; autoscaler provisions one node when a pod targets
`nodeSelector: ci-compile-cpu-hm`.
- New variables: enable_ci_compile_cpu_hm_pool,
ci_compile_cpu_hm_type (default POP2-HM-32C-256G),
ci_compile_cpu_hm_max_size (default 1).
- terragrunt.hcl: enable the pool, max_size=1.
- train-template.yaml: ensure-fxcache nodeSelector pinned to
ci-compile-cpu-hm; memory limit raised 56Gi → 200Gi. The
ci-compile-cpu pool stays as the standard CI compile target for
ensure-binary + every other CPU-heavy task.
Apply with:
cd infra/live/production/kapsule
terragrunt apply -target=module.kapsule.scaleway_k8s_pool.ci_compile_cpu_hm
kubectl apply -n foxhunt -f ../../../../infra/k8s/argo/train-template.yaml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace fragile `git fetch --depth=1 origin "$SHA"` (fails with short SHAs)
with `git clone --no-checkout --filter=blob:none` + `git checkout "$SHA"`
across all 3 Argo templates (5 clone blocks total)
- Upgrade CI compile pool from POP2-32C-128G to POP2-HC-32C-64G
(higher clock EPYC cores, 28% cheaper at €0.851/hr vs €1.18/hr)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename ci-training → ci-training-l40s, add H100/H100x2/H100-SXM/H100-SXM-8
pool resources (all autoscaling 0→1). Remove dev pool (DevPod on platform).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POP2-32C-128G pool now allows up to 4 concurrent nodes for parallel
CI compilation (services + training + builder image builds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN/PPO networks are tiny (3 layers × 128 neurons). Running parallel
hyperopt on GPU wastes cores because CUDA context serializes across
threads — 5 trials on L4 only used 2000m of 6000m requested CPU.
Changes:
- Add --device flag to hyperopt_baseline_rl (auto/cpu/cuda)
- Auto mode forces CPU for parallel runs (no CUDA contention)
- CPU mode uses all available cores (no 2-core reserve)
- Add with_device() builder to DQN/PPO hyperopt trainers
- Downgrade "portfolio value <= 0" and GPU utilization warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split GPU workloads across two pools:
- ci-compile (L4-1-24G): for cargo check/test/build — cheaper, same
CUDA CC 8.9 as L40S, sufficient 24GB VRAM for compilation
- ci-build (L40S-1-48G): for hyperopt + training — 48GB VRAM needed
for large model training batches
Runner defaults to ci-compile; training jobs override to ci-build
via KUBERNETES_NODE_SELECTOR. Both pools autoscale to zero when idle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services moved off L40S GPU node to always-on DEV1-M pool.
Single DEV1-M couldn't fit all services, so enable autoscaling
with max_size=2 to allow a second node when needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove gpu-training (H100) and gpu-inference (L4) pool resources,
variables, and outputs from kapsule module — L40S handles all GPU work
- Switch public-gateway from hardcoded private_network_id to
dependency.kapsule.outputs.private_network_id
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule output wasn't available yet during first gateway apply.
Will switch to dependency.kapsule.outputs.private_network_id after
kapsule re-apply exports it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-XL (48 vCPU) exceeds account quota. GP1-M (16 vCPU, 64GB) fits
within current limits while providing sufficient build resources.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (32 vCPU) creation_error due to project vCPU quota limits.
GP1-M (16 vCPU, 64GB) is 2x the previous GP1-S and within quota.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Autoscaling pool for CI runner pods — scales to zero when idle.
32 cores dramatically improves parallel Rust compilation times.
Also cleaned up the manually-created ci-build-v2 GP1-S pool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.
Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)
Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enable Prometheus server (GitLab chart bundled) on gitlab node pool
- Add socat SSH proxy sidecar (port 2222 → gitlab-shell) to Tailscale proxy
- Remove nginx stream module (not available in alpine) in favor of socat
- Set unlimited nginx client_max_body_size for large git pushes
- Add workhorse extraArgs for API limits
- Explicit always_on_type = DEV1-M in kapsule terragrunt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Terragrunt v0.77+ warns about using terragrunt.hcl as root config.
Rename to root.hcl and update all child includes to use
find_in_parent_folders("root.hcl").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner auto-shuts down after 15min idle (no GITEA-ACTIONS containers).
Gitea server polls every 60s for queued runs and starts instance via SCW API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split single H100 GPU pool into two purpose-specific pools:
- gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
- gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
(observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
(7 services each requesting connection pools exhausted the limit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- kapsule: cluster + 3 node pools (always-on, ci, gpu)
- object-storage: S3 bucket with 30-day sccache expiry
- registry: private Container Registry namespace
- secrets: JWT + DB password via Secret Manager
- block-storage: 100GB b_ssd for training data
All managed via Terragrunt with shared provider/backend.
Gitignore exception added for infra/*/secrets/ (TF code, not actual secrets).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway provider, Object Storage backend for state,
common inputs (region nl-ams, zone nl-ams-1).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
