- Split single H100 GPU pool into two purpose-specific pools:
- gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
- gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
(observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
(7 services each requesting connection pools exhausted the limit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
(pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container
All 10 pods (3 databases + 7 services) now 1/1 Running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
(ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par
Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- train.sh: submit GPU training K8s Jobs with presets
(quick-test, single-model, full-ensemble, hyperopt)
- smoke-test.sh: end-to-end verification of cluster,
networking, databases, services, and node pools
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dockerfile.service: unified multi-stage with sccache S3 support
- Dockerfile.web-gateway: Node dashboard + Rust gateway combined
- Dockerfile.training: CUDA 12.4 with H100 target (compute cap 9.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- kapsule: cluster + 3 node pools (always-on, ci, gpu)
- object-storage: S3 bucket with 30-day sccache expiry
- registry: private Container Registry namespace
- secrets: JWT + DB password via Secret Manager
- block-storage: 100GB b_ssd for training data
All managed via Terragrunt with shared provider/backend.
Gitignore exception added for infra/*/secrets/ (TF code, not actual secrets).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway provider, Object Storage backend for state,
common inputs (region nl-ams, zone nl-ams-1).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
