Persistent block storage for Claude Code, cargo registry, sccache
cache, and shell config. Survives pod restarts and node scale-down.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update COPY paths in all 3 Dockerfiles for crates/bin/services/testing layout
- Migrate service image builds from internal GitLab registry to SCW CR
- Update Kaniko auth to use SCW credentials (nologin + SCW_SECRET_KEY)
- Remove --insecure-registry flags (SCW CR is HTTPS)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redis image WORKDIR is /data and entrypoint runs chown recursively.
PVC mounted at /data/training was read-only, causing chown to fail
with exit code 1, crashing the Redis sidecar.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Mount training-data-pvc at /data/training in build pods via runner config
- Update real_data_loader to search per-symbol subdirectories (Databento layout)
- Support .dbn.zst (zstd-compressed) files alongside raw .dbn
- Add FOXHUNT_DATA_DIR env var override for CI PVC path
- Extract decode_ohlcv_bars helper (generic over reader type)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace pod_spec strategic merge patch (silently not applied) with
runtime_class_name="nvidia". The nvidia RuntimeClass uses the
nvidia-container-runtime which injects GPU drivers, nvidia-smi, and
/dev/nvidia* devices into all containers automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strategic merge patches are applied to the Pod object, not PodSpec.
The patch needs {"spec":{"containers":[...]}} not {"containers":[...]}.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- poll_timeout 180s→600s: H100 provisioning takes ~3-5min from scale-to-zero
- build-ci-builder: only auto-run on push with Dockerfile changes (API
pipelines evaluate changes=true, causing unnecessary kaniko builds)
- check: add needs:[] to decouple from prepare stage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bindgen_cuda calls nvidia-smi to detect compute capability, which fails
without GPU device access. Setting CUDA_COMPUTE_CAP=90 bypasses this for
compilation. Pod spec GPU request ensures NVIDIA device plugin injects
/dev/nvidia* for test-time CUDA execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove orphaned `ci` pool (GP1-XS, 4 vCPU wasted — nothing scheduled to it).
Remove `ci-build` pool (GP1-M) — builds now run on gpu-training (H100-1-80G:
24 vCPU, 240GB, real CUDA). This eliminates the need for CUDA stubs,
separate test-gpu jobs, and ml crate exclusions.
Pool layout after:
always-on DEV1-M (core services, always on)
gitlab GP1-XS (GitLab CE + runner manager, always on)
gpu-training H100-1-80G (CI builds + ML training, scale-to-zero)
gpu-inference L4-1-24G (trading inference, scale-to-zero)
Build pod limits bumped to 16 vCPU / 64GB (from 6/12GB) to use H100 capacity.
Runner now has `gpu` tag — all tests including ml crate run in single job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-XL (48 vCPU) exceeds account quota. GP1-M (16 vCPU, 64GB) fits
within current limits while providing sufficient build resources.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (32 vCPU) creation_error due to project vCPU quota limits.
GP1-M (16 vCPU, 64GB) is 2x the previous GP1-S and within quota.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Autoscaling pool for CI runner pods — scales to zero when idle.
32 cores dramatically improves parallel Rust compilation times.
Also cleaned up the manually-created ci-build-v2 GP1-S pool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate links libcuda.so.1 at runtime (via candle CUDA).
GPU-less CI nodes need the stub library so binaries can load, but
model_loader tests that actually use CUDA must be skipped. These will
run on gpu-training nodes separately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The backtesting crate dynamically links to libcuda.so.1 via candle.
NVIDIA CUDA dev images include stubs at /usr/local/cuda/lib64/stubs/
but only as libcuda.so (not .so.1). Create the symlink both in the
Dockerfile (for future builds) and inline in the test script (for
the current image).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gitea replaced by GitLab CE on Kapsule. CI runner replaced by
GitLab Runner with Kubernetes executor on ci-build node pool.
Removed:
- .gitea/workflows/ci.yaml (Gitea Actions config)
- infra/modules/ci-runner/ (Scaleway VM-based runner)
- infra/live/production/ci-runner/ (Terragrunt live config)
Note: The ci-runner VM instance still needs manual `terragrunt destroy`
before the Scaleway resource can be released.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add .gitignore entries for .dbn, .safetensors, .onnx and other large binaries
- Point GitLab Runner at internal cluster URL (not external Tailscale)
- Disable nginx proxy_buffering for git clone/push through Tailscale proxy
- Increase socat SSH proxy buffer sizes to 1MB
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Enable Prometheus server (GitLab chart bundled) on gitlab node pool
- Add socat SSH proxy sidecar (port 2222 → gitlab-shell) to Tailscale proxy
- Remove nginx stream module (not available in alpine) in favor of socat
- Set unlimited nginx client_max_body_size for large git pushes
- Add workhorse extraArgs for API limits
- Explicit always_on_type = DEV1-M in kapsule terragrunt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Terragrunt v0.77+ warns about using terragrunt.hcl as root config.
Rename to root.hcl and update all child includes to use
find_in_parent_folders("root.hcl").
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 15min timeout was too short for Rust builds. Also checks
for act_runner child processes (image pull, git clone phases)
not just Docker containers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner auto-shuts down after 15min idle (no GITEA-ACTIONS containers).
Gitea server polls every 60s for queued runs and starts instance via SCW API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix container registry region nl-ams → fr-par across all CI jobs
- Add sccache build-args to training image build (no-op fallback for local)
- Add rclone to training Docker runtime for S3 output sync
- Update train.sh: S3 sync on Job completion via rclone env-var config,
--run-id tracking, evaluate preset for walk-forward evaluation
- Add s3-credentials Secret template (.example, apply via kubectl)
- Add design doc and implementation plan
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split single H100 GPU pool into two purpose-specific pools:
- gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
- gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
(observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
(7 services each requesting connection pools exhausted the limit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
(pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container
All 10 pods (3 databases + 7 services) now 1/1 Running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove deleted foxhunt-deploy COPY from all Dockerfiles, update sccache
default region to fr-par, add build-and-push.sh for building all service
images and pushing to rg.fr-par.scw.cloud/foxhunt/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- K8s API ACL restricted to Tailscale CGNAT (100.64.0.0/10) + admin IPs
- CI and GPU pools: min_size=0 with lifecycle ignore_changes on size
so autoscaler manages actual node count without Terraform drift
- Add harden.sh script for post-provisioning security lockdown
(ACL setup, exposed service check, bucket ACL verify, registry visibility)
- Update smoke-test.sh default region to fr-par
Security audit results:
- All services ClusterIP only (no LoadBalancer/NodePort)
- No Ingress resources
- Container registry: private
- S3 buckets: HTTP 403 on anonymous access
- Gitea: all ports closed on public IP (Tailscale-only via security group)
- Kapsule nodes: no open ports on external IPs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- train.sh: submit GPU training K8s Jobs with presets
(quick-test, single-model, full-ensemble, hyperopt)
- smoke-test.sh: end-to-end verification of cluster,
networking, databases, services, and node pools
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Dockerfile.service: unified multi-stage with sccache S3 support
- Dockerfile.web-gateway: Node dashboard + Rust gateway combined
- Dockerfile.training: CUDA 12.4 with H100 target (compute cap 9.0)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
