Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pod-level runAsNonRoot:true conflicts with initContainer runAsUser:0.
Kubernetes rejects the container. Add explicit override so the
initContainer can still run as root to fetch binaries from S3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
default-deny and infrastructure ingress rules match correctly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh.
Also updated references in postgres.yaml, migrate.yaml,
and postgres-init.yaml to use db-credentials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable GitLab Prometheus to auto-discover and scrape metrics from all
foxhunt pods by adding gitlab.com/prometheus_scrape annotations to the
pod template metadata of 9 service deployments and the training job
template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Disable file logging in init_observability — use console-only (stdout).
Containerd captures stdout, Promtail ships to Loki. Eliminates
"Failed to create log directory: Permission denied" errors on 3 services.
- Remove LOG_DIR env vars and /app/logs emptyDir volumes from 5 deployments.
- Add OTEL_EXPORTER_OTLP_ENDPOINT=tempo to 5 services that were missing it
(api-gateway, web-gateway, data-acquisition, trading-agent, broker-gateway).
- Make OTLP tracing init non-fatal — services degrade gracefully if Tempo is
unavailable instead of failing entire observability stack.
- Stop web-gateway gRPC stream retry on Unimplemented status. When trading-service
doesn't implement streaming endpoints, log once and stop instead of retrying
every 60s indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init_observability now reads LOG_DIR env var for log directory (falls back
to relative logs/ path). Gracefully disables file logging if directory
cannot be created instead of failing the entire observability init.
- Add ServiceAccount, Role, and RoleBinding for ml-training-service to
create/get/list/watch/delete batch/v1 Jobs (K8s training dispatch).
- Add LOG_DIR=/app/logs env to ml-training-service, trading-service, and
backtesting-service deployment YAMLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime base image runs as uid 1000 (foxhunt user), but
new PVC volumes are owned by root. The initContainer needs root to
write the cached binary to the PVC. Main container still runs as
foxhunt (non-root) for security.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously nodeSelector was only applied via kubectl patch at deploy time.
Now it's in the YAML files to keep git and cluster in sync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep existing git.fxhnt.ai hostname for GitLab CE (repoint DNS
to new Tailscale IP after deploy). Add Terraform DNS module to
manage the A record via Terragrunt instead of manual scw CLI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split single H100 GPU pool into two purpose-specific pools:
- gpu-training: H100-1-80G (€2.73/hr) for 10-model ensemble training
- gpu-inference: L4-1-24G (€0.75/hr) for cost-effective trading inference
- Add GPU taint controller DaemonSet that auto-taints new GPU nodes
with nvidia.com/gpu=true:NoSchedule to prevent non-GPU workloads
- Fix service log directory permissions with emptyDir volumes
(observability init fails creating /app/logs as non-root user)
- Increase postgres max_connections from 25 to 100
(7 services each requesting connection pools exhausted the limit)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DB pool connect/acquire timeouts (50ms→5s) for cluster networking
(pool-level timeouts, not query timeouts — HFT query timeout stays at 800μs)
- Fix secret key references (DATABASE_PASSWORD→db-password) in all manifests
- Fix api-gateway port (50050→50051) to match actual gRPC listen port
- Fix web-gateway health probe path (/api/health→/health)
- Fix S3 endpoint region (nl-ams→fr-par) in ml-training-service
- Add TLS cert volume mount for ml-training-service
- Add BENZINGA_API_KEY placeholder for backtesting-service startup
- Remove always-on nodeSelector from services (let autoscaler handle)
- Add serve subcommand to ml-training-service container
All 10 pods (3 databases + 7 services) now 1/1 Running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
