containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic deploy job with per-service deploy jobs that fire
as soon as each compile finishes. No more waiting for all 8 services
+ training to compile before any deployment starts.
Structure:
- .deploy-base: shared kubeconfig setup
- .deploy-service-base: upload binary + apply manifest + rollout restart
- deploy-<service> (×8): needs compile-<service>, same rules via YAML anchors
- deploy-training: uploads training binaries after compile-training
- deploy-web-dashboard: uploads web assets after build-web-dashboard
- deploy-infra: migrations, monitoring, dashboards, proxy (no compile dep)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When all 8 per-service compile jobs fire simultaneously (e.g. on
Cargo.toml change), the single POP2-32C-128G node is fully loaded.
Training compile pod can't schedule until service jobs finish.
KUBERNETES_POLL_TIMEOUT=1200 gives it time to wait.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each service gets its own compile job with dependency-aware changes:
filters so only affected services recompile on push. Deploy job
selectively restarts only services whose binary was compiled.
- Add .compile-service-base template (4 CPUs, 8Gi per job)
- 8 jobs: api-gateway, trading-service, ml-training-service,
backtesting-service, trading-agent-service, broker-gateway,
data-acquisition-service, web-gateway
- Deploy needs: all 8 compile jobs (optional: true, artifacts: true)
- Rollout restart loop checks build-out/services/ for actual binaries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training job before_script now fetches from MinIO S3 when CI
artifacts are unavailable. Fixes hyperopt/train jobs failing with
"command not found" when triggered without compile-training artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CPU request 14000m→10000m (burstable to 15500m), CARGO_BUILD_JOBS 14→10.
Three compile jobs now total 22000m on 31800m node (was 30000m).
Per-job resource_group prevents cross-pipeline contention.
interruptible: true lets newer pipelines cancel stale ones.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clear separation: l40s tag → L40S pool, h100 tag → H100 pool.
Updated runner-rl-values.yaml and CI base templates to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ci-training-h100 runner (runner-h100-values.yaml)
- Route hyperopt-dqn and hyperopt-ppo to H100 via kapsule,h100 tags
- H100 is ~2x faster and cheaper per run vs L40S for hyperopt workloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
evaluate_baseline was using default hidden_dim_base=128 instead of the
hyperopt-optimized 4096, causing checkpoint loading to fail silently
(tensor shape mismatch). This produced empty folds: [] in eval reports.
Also adds consistent --tx-cost-bps/--tick-size/--spread-ticks to
train_baseline_rl and evaluate_baseline in the hyperopt-dqn job so all
three pipeline stages (hyperopt, train, eval) use the same cost model.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.
Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
reference api_gateway (not monitoring_service) as training metrics host
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-services job already builds monitoring_service but the deploy
job was missing it from the PVC copy loop and rollout restart/wait loops.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The auto-detect heuristic used cpus/2 ("smart cap"), designed for
CPU-bound workloads. DQN/PPO trials are GPU-bound — each rayon
thread submits CUDA kernels and waits on cudaDeviceSynchronize(),
using minimal CPU. cpus-1 is the correct cap.
On L40S-1-48G (8 vCPU): 3 threads → 7 threads (2.3× more trials).
Also bumps CI CPU limit 7500m→8000m to expose all 8 cores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3→PVC migration had several issues causing deploy failures:
- Stale binary-writer pod from previous failed deploys blocked new ones
(terminated pods can't be updated via kubectl apply)
- Services were applied BEFORE binaries written to PVC, so first deploy
or empty PVC caused pods to crash (binary not found)
- GPU overlay files in services/ dir were auto-applied by kubectl apply,
referencing nonexistent S3/minio secrets and PVCs
- Training job template mounted foxhunt-binaries PVC but training runs
on ci-training node — RWO PVC is bound to foxhunt node
Fixes:
- Reorder deploy: write binaries → apply manifests → rollout restart
- Clean up stale writer pods before creating new ones
- Move GPU overlays to gpu-overlays/ (manual apply only), update to PVC
- Add training-binaries PVC for GPU node, best-effort population
- Training job template uses initContainer to copy from training PVC
- set -e for critical path, set +e for optional training binary copy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The before_script was copying training binaries to ${CI_PROJECT_DIR}/bin/
which collides with the git checkout bin/fxt/ directory. chmod +x on the
fxt subdirectory fails with "Operation not permitted", crashing the job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner #2 already has tags [kapsule, gpu] but CI jobs still used
kapsule-rl. Also removes stale minio-ca-cert volume from runner values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy stage needs compile-training artifacts to copy training binaries
to the foxhunt-binaries PVC alongside service binaries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MinIO pod was removed but compile-training still uploaded to it via rclone,
causing 403 Forbidden failures. Now:
- compile-training: saves binaries as CI artifacts (3 day TTL)
- deploy stage: copies training binaries to foxhunt-binaries PVC via binary-writer
- CI training jobs: use compile-training CI artifacts directly
- job-template.yaml: mounts foxhunt-binaries PVC (no initContainer)
- Removed MinIO deploy from deploy stage (no longer needed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change training job tags from kapsule to kapsule-rl so they are
picked up exclusively by the RL runner (GPU-dedicated), not the
general runner (CPU compile pool).
- Add OTEL_EXPORTER_OTLP_ENDPOINT to both .train-rl-base and
.train-validate-base so training binaries export OTLP traces
to Tempo.
- Update allow-monitoring-scrape NetworkPolicy to include
foxhunt-ci pods so CI training pods can reach Tempo:4317.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix tried appending to /etc/ssl/certs/ca-certificates.crt
which fails on non-root containers (Permission denied). Instead,
copy system CAs to /tmp/ca-bundle.crt, append MinIO CA, and set
SSL_CERT_FILE for Go's crypto/tls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RCLONE_CA_CERT env var does not apply to rclone's S3 backend — Go's
crypto/tls reads from the system CA bundle. Append the MinIO CA cert
to /etc/ssl/certs/ca-certificates.crt before rclone commands. Fixes
compile-services, compile-training, web-dashboard, write-manifest,
and both training base templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire BF16/FP16 mixed precision end-to-end for DQN and PPO with auto-detection
from GPU name (Ampere+ → BF16, Volta/Turing → FP16). Add hidden_dim_base to
hyperopt and wire through training/eval binaries. Reduce GPU sync points: make
DQN NaN checks periodic (every 100 steps), replace PPO GAE GPU round-trip with
pure CPU implementation. Cache training data across hyperopt trials for all 10
models via Arc. Batch DQN experience storage (128x fewer lock acquisitions).
Correct VRAM constants and batch bounds for all 9 supervised model adapters.
28 files changed, +1207/-208 lines. 2418 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner expects KUBERNETES_NODE_SELECTOR_<NAME>: "key=value" format,
not KUBERNETES_NODE_SELECTOR_<key>: "value". Fix both compile (ci-compile-cpu)
and supervised training (ci-training) node selector overrides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
.rust-base-cpu was missing KUBERNETES_NODE_SELECTOR override, so compile
jobs defaulted to the runner's ci-rl pool which has scale-to-zero and
timed out. ci-compile-cpu has a persistent POP2-32C-128G node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job's Grafana dashboard ConfigMap creation used `declare -A`
associative arrays which require bash 4+. The CI runner's shell is
/bin/sh (dash), causing "cannot convert indexed to associative array"
errors and failing the deploy stage.
Rewrites the logic as a POSIX-compatible shell function that takes the
ConfigMap name, folder, and dashboard files as positional arguments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.
- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format
IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ohlcv_bars table migration (050) for trading_service prediction loop
- Fix kube-state-metrics RBAC: add admissionregistration.k8s.io and
certificates.k8s.io API groups to suppress forbidden errors
- Fix web-gateway gRPC stream reconnect: detect Unimplemented via both
status code and error message text (proxy may remap codes)
- Add automated migration step to CI deploy pipeline with tracking table
(_applied_migrations) so migrations are applied before service rollouts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
