The real gateway foxhunt-gw (42653780) was an unmanaged orphan: kapsule state never had a gateway,
public-gateway state referenced a deleted gateway (7a7ffaa5) -> BOTH modules' terragrunt apply would
have created a duplicate gateway (risking cluster egress). Fix:
- kapsule module: removed the legacy gateway/dhcp/gateway_network resources (pre-refactor duplicate,
never in kapsule state; cluster only needs the private_network which stays). kapsule plan now clean.
- public-gateway module (canonical, IPAM mode): state rm the stale 7a7ffaa5 refs + imported the real
gateway/gateway_network/IP (42653780 / 661c8543 / dc419a10); set bastion_enabled=true to match the
live gw. public-gateway plan now 'No changes'. Live gateway untouched throughout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reasons:
- h100-sxm: Scaleway account quota is 0/0 for the SXM instance type
(cp_servers_type_H100_SXM_2_80G). The pool was perpetually trying
to maintain size=1 with a creation_error node, which JAMMED the
cluster autoscaler. That blocked L40S scale-up entirely during
today's alpha-perception cluster run.
- h100x2: more expensive than current workloads justify. Every
training path (alpha perception, future PPO) fits on either the
single-GPU H100 or L40S.
Removed:
- Two `scaleway_k8s_pool` resources from infra/modules/kapsule/main.tf
- Six variables (enable + type + max_size for each pool)
- Two outputs (pool_id for each)
- Corresponding inputs in infra/live/production/kapsule/terragrunt.hcl
The live cluster has the SXM pool stuck node manually deleted via
`scw k8s node delete` (this commit-session); the pool resource itself
will be destroyed on next `terragrunt apply`.
Post-cleanup pool inventory:
- platform (DEV1-L × 3)
- ci-training-h100 (H100-1-80G, max 1) <- regular single-GPU
- ci-training-l40s (L40S-1-48G, max 1) <- primary training target
- ci-compile-cpu (POP2-HC, max 4)
- ci-compile-cpu-hm (POP2-HM, max 1)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two 9-quarter precompute_features runs OOM-killed on the existing
ci-compile-cpu pool (POP2-HC-32C-64G, 56Gi cgroup limit):
- train-wq8b8: exit 137 ~12s after "OFI computed" at ~57Gi
- train-2l6p4: exit 137 ~12s after "OFI computed" at ~52Gi peak
(despite the into_iter + drop(feature_vectors) +
normalize_in_place refactors landed in a27cb40a9 + 623ebfcf7,
which trimmed ~12GB of avoidable retention)
The remaining ~52-60GB peak is the irreducible working set at the
post-OFI / pre-alpha-pipeline step:
- 9-quarter MBP-10 snapshots (~10GB)
- 199M front-month trades as Mbp10Trade (~13GB)
- 17.8M bars × features + targets + OFI buffers (~14GB)
- alpha_snapshots (move-handoff from all_snapshots, ~10GB)
- feature/target Vecs (~7GB) + Rust allocator overhead
Adds a dedicated high-memory pool sized for this rare path:
- New scaleway_k8s_pool.ci_compile_cpu_hm (POP2-HM-32C-256G,
32 vCPU + 256GB RAM) with size=0 + min_size=0 autoscaling. Costs
zero when idle; autoscaler provisions one node when a pod targets
`nodeSelector: ci-compile-cpu-hm`.
- New variables: enable_ci_compile_cpu_hm_pool,
ci_compile_cpu_hm_type (default POP2-HM-32C-256G),
ci_compile_cpu_hm_max_size (default 1).
- terragrunt.hcl: enable the pool, max_size=1.
- train-template.yaml: ensure-fxcache nodeSelector pinned to
ci-compile-cpu-hm; memory limit raised 56Gi → 200Gi. The
ci-compile-cpu pool stays as the standard CI compile target for
ensure-binary + every other CPU-heavy task.
Apply with:
cd infra/live/production/kapsule
terragrunt apply -target=module.kapsule.scaleway_k8s_pool.ci_compile_cpu_hm
kubectl apply -n foxhunt -f ../../../../infra/k8s/argo/train-template.yaml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Scaleway Kapsule GPU nodes need NVIDIA driver DKMS compile on cold
start (~15 min). Setting min_size=1 keeps the node warm between runs.
Scale to 0 manually when done training for the day.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added scaleway_vpc_public_gateway + DHCP + gateway_network to TF
(was manually created, now codified with push_default_route=true)
- Added scripts/safe-node-replace.sh — one-at-a-time with DNS verification
- Added dns-bootstrap-policy.yaml — incident documentation + recovery procedure
- Bastion enabled (port 61000) for emergency SSH access
Prevention: NEVER replace all nodes at once. Use safe-node-replace.sh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove prometheus_fqdn and monitor_fqdn outputs that referenced
deleted DNS records. Add chat_fqdn and mail_fqdn for Mattermost
and Stalwart services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Mattermost Team Edition deployment (PVC, Service, NetworkPolicy),
DNS records (chat.fxhnt.ai, mail.fxhnt.ai), Tailscale nginx proxy
server block with WebSocket upgrade support, and Mattermost egress
rules to all 5 Argo workflow NetworkPolicies for exit handler webhooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add terragrunt-apply step to Argo CI pipeline (plan+apply on main push
when infra/live/ or infra/modules/ change)
- Bake OpenTofu 1.9.0 + Terragrunt 0.77.12 into ci-builder-cpu image
with SHA256 checksum verification
- Remove 3 ghost node pools (foxhunt, gitlab, h100-sxm8) from kapsule
module to match Scaleway reality
- Make terragrunt.hcl single source of truth (remove variable defaults)
- Fix GitLab TF state lock methods (POST/DELETE for HTTP backend)
- Harden PAT rotation: more retries, verification step, recovery docs
- Add weekly PAT expiry check CronJob (warns 14 days before expiry)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace fragile `git fetch --depth=1 origin "$SHA"` (fails with short SHAs)
with `git clone --no-checkout --filter=blob:none` + `git checkout "$SHA"`
across all 3 Argo templates (5 clone blocks total)
- Upgrade CI compile pool from POP2-32C-128G to POP2-HC-32C-64G
(higher clock EPYC cores, 28% cheaper at €0.851/hr vs €1.18/hr)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename ci-training → ci-training-l40s, add H100/H100x2/H100-SXM/H100-SXM-8
pool resources (all autoscaling 0→1). Remove dev pool (DevPod on platform).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule doesn't allow node-labels in kubelet_args. Instead, use the
auto-assigned k8s.scaleway.com/pool-name label that every node gets.
- Revert kubelet_args from Terraform (not supported)
- Switch runner node_selector: pool→k8s.scaleway.com/pool-name
- Update CI pipeline KUBERNETES_NODE_SELECTOR to match
- Helm upgraded both runners (CPU + GPU)
No more manual kubectl label after node scale-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nodes from autoscaler now get pool=<name> labels automatically via
kubelet_args, eliminating the need to manually label nodes after
scale-up events.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split GPU workloads across two pools:
- ci-compile (L4-1-24G): for cargo check/test/build — cheaper, same
CUDA CC 8.9 as L40S, sufficient 24GB VRAM for compilation
- ci-build (L40S-1-48G): for hyperopt + training — 48GB VRAM needed
for large model training batches
Runner defaults to ci-compile; training jobs override to ci-build
via KUBERNETES_NODE_SELECTOR. Both pools autoscale to zero when idle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services moved off L40S GPU node to always-on DEV1-M pool.
Single DEV1-M couldn't fit all services, so enable autoscaling
with max_size=2 to allow a second node when needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove gpu-training (H100) and gpu-inference (L4) pool resources,
variables, and outputs from kapsule module — L40S handles all GPU work
- Switch public-gateway from hardcoded private_network_id to
dependency.kapsule.outputs.private_network_id
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scale-to-zero pools start with 0 nodes, so Terraform's default
wait_for_pool_ready=true always fails with "state warning, wants ready".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.
Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)
Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
