- Disable file logging in init_observability — use console-only (stdout).
Containerd captures stdout, Promtail ships to Loki. Eliminates
"Failed to create log directory: Permission denied" errors on 3 services.
- Remove LOG_DIR env vars and /app/logs emptyDir volumes from 5 deployments.
- Add OTEL_EXPORTER_OTLP_ENDPOINT=tempo to 5 services that were missing it
(api-gateway, web-gateway, data-acquisition, trading-agent, broker-gateway).
- Make OTLP tracing init non-fatal — services degrade gracefully if Tempo is
unavailable instead of failing entire observability stack.
- Stop web-gateway gRPC stream retry on Unimplemented status. When trading-service
doesn't implement streaming endpoints, log once and stop instead of retrying
every 60s indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init_observability now reads LOG_DIR env var for log directory (falls back
to relative logs/ path). Gracefully disables file logging if directory
cannot be created instead of failing the entire observability init.
- Add ServiceAccount, Role, and RoleBinding for ml-training-service to
create/get/list/watch/delete batch/v1 Jobs (K8s training dispatch).
- Add LOG_DIR=/app/logs env to ml-training-service, trading-service, and
backtesting-service deployment YAMLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway quota limits POP2-32C-128G to 1 instance. Reduce each compile
job from 28→14 CPUs so compile-services and compile-training run in
parallel on the same 32-vCPU node instead of sequentially.
Also fix data-acquisition-service health probe: service doesn't implement
gRPC health protocol, switch to tcpSocket probe on port 50057.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime base image runs as uid 1000 (foxhunt user), but
new PVC volumes are owned by root. The initContainer needs root to
write the cached binary to the PVC. Main container still runs as
foxhunt (non-root) for security.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway block storage minimum volume size is 1Gi. The 500Mi PVCs
were stuck in Pending state, blocking all service pod scheduling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web dashboard build now uses node:22-slim with inline rclone install
instead of a custom Kaniko-built image. This removes 1 Kaniko build
from the prepare stage, reducing pod scheduling contention.
Only 2 runtime images needed: CPU (foxhunt-runtime) and GPU
(foxhunt-training-runtime). Binary selection is via K8s command.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collection kernels (dqn_experience_kernel.cu,
ppo_experience_kernel.cu) use cudarc::nvrtc::compile_ptx() at runtime.
Without libnvrtc.so the kernel compile fails silently and falls back to
CPU experience collection. Adding cuda-nvrtc-12-4 (~30MB) enables full
GPU-accelerated experience collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both images now use identical cuda:12.4.1-cudnn-runtime base, so maintaining
two separate Dockerfiles and build jobs was wasteful. Single image contains
all 7 training binaries, halving registry storage and Kaniko build time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
candle's CUDA backend links against libcudnn.so.9 for kernel operations.
Switch training-rl base from cuda:12.4.1-runtime to cuda:12.4.1-cudnn-runtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL models (DQN/PPO) only need basic CUDA runtime (~2GB), while supervised
models need cuDNN (~4GB). Splitting saves ~2GB pull time per RL job.
Rename the L4 GPU pool from ci-compile to ci-rl to reflect its actual use.
Add DaemonSet image pre-puller to cache training images on GPU nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-services stage outputs binaries to build-out/services/ but
the web-gateway Dockerfile still referenced the old build-out/ path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POP2-32C-128G pool now allows up to 4 concurrent nodes for parallel
CI compilation (services + training + builder image builds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Based on rust:1.89-slim-bookworm (~2-3GB vs ~8GB CUDA devel).
Same toolchain: mold 2.35, protoc 28.3, sccache 0.10, clang, lld.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN/PPO networks are tiny (3 layers × 128 neurons). Running parallel
hyperopt on GPU wastes cores because CUDA context serializes across
threads — 5 trials on L4 only used 2000m of 6000m requested CPU.
Changes:
- Add --device flag to hyperopt_baseline_rl (auto/cpu/cuda)
- Auto mode forces CPU for parallel runs (no CUDA contention)
- CPU mode uses all available cores (no 2-core reserve)
- Add with_device() builder to DQN/PPO hyperopt trainers
- Downgrade "portfolio value <= 0" and GPU utilization warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule doesn't allow node-labels in kubelet_args. Instead, use the
auto-assigned k8s.scaleway.com/pool-name label that every node gets.
- Revert kubelet_args from Terraform (not supported)
- Switch runner node_selector: pool→k8s.scaleway.com/pool-name
- Update CI pipeline KUBERNETES_NODE_SELECTOR to match
- Helm upgraded both runners (CPU + GPU)
No more manual kubectl label after node scale-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nodes from autoscaler now get pool=<name> labels automatically via
kubelet_args, eliminating the need to manually label nodes after
scale-up events.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changed from non-existent scw-bssd-nfs to scw-bssd (RWO).
Training output is single-pod, doesn't need ReadWriteMany.
PVC deployed to foxhunt namespace (WaitForFirstConsumer).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL env simulation is single-threaded (~1 core actual). Lowering from
7000m to 2000m request allows DQN + PPO hyperopt to run simultaneously
on one L4 node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix ServiceAccount: use serviceAccount.name (not deprecated serviceAccountName)
- concurrent=1: one RL job gets full L4 node (7000m/7800m CPU, 16Gi/40Gi RAM)
- Faster iteration: ~2.3x more CPU per job vs splitting across 2 concurrent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set runtime_class_name=nvidia at runner level (all GPU pools have GPUs)
- Fix KUBERNETES_NODE_SELECTOR format: "key=value" not just value
- Replace wrong *_overwrite_allowed (regex) with correct
*_overwrite_max_allowed (max value) for CPU/memory overrides
- Remove per-job KUBERNETES_RUNTIME_CLASS_NAME (runner-level handles it)
- Simplify GPU detection in test/compile (GPU always present now)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-services OOMKilled at 8Gi — ml crate with CUDA needs ~20Gi.
CI variable overrides (KUBERNETES_MEMORY_LIMIT) are silently dropped
by runner config merger for non-auto-generated fields. Increase
defaults to 3500m/12Gi request, 7800m/28Gi limit to fit L4 nodes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Setting runtime_class_name = "" causes Kubernetes to reject pod creation
with "resource name may not be empty". Omitting it entirely means no
runtimeClassName on pod spec by default; GPU jobs override via
KUBERNETES_RUNTIME_CLASS_NAME CI variable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
L4 ci-compile node was failing with 'no runtime for nvidia' because
fresh nodes take time to install nvidia drivers. Kaniko builds don't
need GPU at all. Now only Rust compile/test and training jobs request
nvidia RuntimeClass via KUBERNETES_RUNTIME_CLASS_NAME CI variable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- compile-services now depends on check (not test), running in parallel
with test stage — saves ~10 min from critical path
- Runner concurrent bumped 4→10 so all 9 Kaniko builds run simultaneously
- Runner default resources lowered (500m/1Gi) for lightweight Kaniko jobs;
Rust compile/test/training override via KUBERNETES_*_REQUEST CI vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously nodeSelector was only applied via kubectl patch at deploy time.
Now it's in the YAML files to keep git and cluster in sync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s nodeSelectors, CI config, runner config, monitoring docs, and
smoke tests now reference the new pool names provisioned by terragrunt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split GPU workloads across two pools:
- ci-compile (L4-1-24G): for cargo check/test/build — cheaper, same
CUDA CC 8.9 as L40S, sufficient 24GB VRAM for compilation
- ci-build (L40S-1-48G): for hyperopt + training — 48GB VRAM needed
for large model training batches
Runner defaults to ci-compile; training jobs override to ci-build
via KUBERNETES_NODE_SELECTOR. Both pools autoscale to zero when idle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Services moved off L40S GPU node to always-on DEV1-M pool.
Single DEV1-M couldn't fit all services, so enable autoscaling
with max_size=2 to allow a second node when needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove gpu-training (H100) and gpu-inference (L4) pool resources,
variables, and outputs from kapsule module — L40S handles all GPU work
- Switch public-gateway from hardcoded private_network_id to
dependency.kapsule.outputs.private_network_id
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
