Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule doesn't allow node-labels in kubelet_args. Instead, use the
auto-assigned k8s.scaleway.com/pool-name label that every node gets.
- Revert kubelet_args from Terraform (not supported)
- Switch runner node_selector: pool→k8s.scaleway.com/pool-name
- Update CI pipeline KUBERNETES_NODE_SELECTOR to match
- Helm upgraded both runners (CPU + GPU)
No more manual kubectl label after node scale-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nodes from autoscaler now get pool=<name> labels automatically via
kubelet_args, eliminating the need to manually label nodes after
scale-up events.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split GPU workloads across two pools:
- ci-compile (L4-1-24G): for cargo check/test/build — cheaper, same
CUDA CC 8.9 as L40S, sufficient 24GB VRAM for compilation
- ci-build (L40S-1-48G): for hyperopt + training — 48GB VRAM needed
for large model training batches
Runner defaults to ci-compile; training jobs override to ci-build
via KUBERNETES_NODE_SELECTOR. Both pools autoscale to zero when idle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services moved off L40S GPU node to always-on DEV1-M pool.
Single DEV1-M couldn't fit all services, so enable autoscaling
with max_size=2 to allow a second node when needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove gpu-training (H100) and gpu-inference (L4) pool resources,
variables, and outputs from kapsule module — L40S handles all GPU work
- Switch public-gateway from hardcoded private_network_id to
dependency.kapsule.outputs.private_network_id
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scale-to-zero pools start with 0 nodes, so Terraform's default
wait_for_pool_ready=true always fails with "state warning, wants ready".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.
Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)
Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 15min timeout was too short for Rust builds. Also checks
for act_runner child processes (image pull, git clone phases)
not just Docker containers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner auto-shuts down after 15min idle (no GITEA-ACTIONS containers).
Gitea server polls every 60s for queued runs and starts instance via SCW API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split single H100 GPU pool into two purpose-specific pools:
- gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
- gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
(observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
(7 services each requesting connection pools exhausted the limit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
(ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par
Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- kapsule: cluster + 3 node pools (always-on, ci, gpu)
- object-storage: S3 bucket with 30-day sccache expiry
- registry: private Container Registry namespace
- secrets: JWT + DB password via Secret Manager
- block-storage: 100GB b_ssd for training data
All managed via Terragrunt with shared provider/backend.
Gitignore exception added for infra/*/secrets/ (TF code, not actual secrets).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
