The remoteWrite block had placeholder values (COCKPIT_METRICS_PUSH_URL,
COCKPIT_PUSH_TOKEN) that were never replaced, causing Prometheus to
spam "Failed to send batch" warnings every minute. Cockpit was replaced
by self-hosted Grafana+Prometheus+Loki+Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Readiness/liveness probes were checking port 4004 (socat), which
always listens even when the gateway is stuck at login — masking
failures. Now probes check port 4002 (TWS API), which only opens
after successful IBKR authentication.
Deploy strategy changed to Recreate because IBKR allows only one
session per account. RollingUpdate starts the new pod before killing
the old one, causing both to fight over the session in a crash loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gnzsnz/ib-gateway image renders its IBC config.ini from a template
using envsubst. The ExistingSessionDetectedAction field was rendering
as empty, causing IBC to show a GUI dialog when IBKR detected a
competing session — which nobody could dismiss in a headless pod.
Setting primaryoverride tells IBC to automatically take over any
existing session, which is the correct behavior for a K8s deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule autoscaler already handles GPU node scale-down
(10min idle timeout). This CronJob only printed a message
and served no purpose.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana now stores users, preferences, and state in our existing
PostgreSQL instance (dedicated 'grafana' database). This eliminates
the 2Gi PVC dependency — all persistent state lives in Postgres,
dashboards in ConfigMaps, datasources in Helm values.
Also adds Grafana to Postgres NetworkPolicy ingress allowlist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana PVC was lost during cleanup (orphaned PV deletion).
Dashboards survived (ConfigMap-based), but user accounts and
datasources were lost.
This commit captures all Helm values previously set via --set:
- Sidecar config with foldersFromFilesStructure for folder organization
- Loki + Tempo datasources alongside Prometheus
- Platform node toleration for gitlab taint
Also fixed CI/CD folder annotation (slash caused nested directory).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI executor pods (label foxhunt-ci) were selected by allow-monitoring-scrape
(which uses matchExpressions In [foxhunt, foxhunt-ci]) making them
policy-controlled, but allow-dns only selected foxhunt pods via matchLabels.
This blocked DNS resolution → git clone failure in deploy jobs.
- Update allow-dns podSelector to matchExpressions In [foxhunt, foxhunt-ci]
- Add ci-egress policy granting broad egress for ephemeral CI pods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Infrastructure Cockpit:
- GPU gauges → side-by-side timeseries (GPU Utilization + GPU VRAM with total line)
- Added Memory by Service timeseries next to existing CPU by Service
- Cluster resource gauges shrunk to h=4 for tighter layout
Training Cockpit:
- GPU Utilization/Memory gauges → side-by-side timeseries matching infra cockpit
- GPU Temperature/Power widened from w=6 to w=12 in paired row
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 stat panels (Ensemble Agreement, Prediction Confidence, Model Accuracy,
Ensemble Sharpe, Realized PnL, Unrealized PnL) showed red when ML metrics
don't exist yet. Added special null mapping → "Inactive" in dark-blue.
Panels will still show red/yellow/green when the trading system is active
and emitting metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Active CI Jobs" stat panel used sum(gitlab_ci_active_jobs_sum) which
is a histogram accumulator that only grows (was showing 982 red). Changed
to sum(increase(pipeline_processing_events_total[1h])) which shows actual
pipeline activity in the last hour. Threshold changed to blue=idle,
green=active.
Also fixed Grafana provisioning:
- Removed overlapping named providers (cockpits/operations/infrastructure/cicd)
that duplicated sidecarProvider, causing UID conflicts and write lockout
- Enabled foldersFromFilesStructure on sidecarProvider
- Unlabeled grafana-dashboards-foxhunt ConfigMap (duplicate of cockpits+operations)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change training job tags from kapsule to kapsule-rl so they are
picked up exclusively by the RL runner (GPU-dedicated), not the
general runner (CPU compile pool).
- Add OTEL_EXPORTER_OTLP_ENDPOINT to both .train-rl-base and
.train-validate-base so training binaries export OTLP traces
to Tempo.
- Update allow-monitoring-scrape NetworkPolicy to include
foxhunt-ci pods so CI training pods can reach Tempo:4317.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI build pods have label app.kubernetes.io/part-of=foxhunt-ci,
not foxhunt. Add foxhunt-ci to the MinIO NetworkPolicy ingress
values so compile/upload jobs can reach MinIO S3 endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New row shows GitLab, CI Runners, Monitoring, Collectors, Tailscale,
and Platform Total stat panels, plus a pod status table, CPU/memory
timeseries, and container restart chart for all non-foxhunt pods.
Also fix minio-init-buckets job: remove part-of:foxhunt label (triggers
default-deny egress block) and rely on minio NetworkPolicy ingress
allowlist for minio-init pods instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The minio-init-buckets job pod has label app.kubernetes.io/name: minio-init
which doesn't match part-of: foxhunt (adding part-of would trigger
default-deny egress). Add explicit ingress allowlist entry for minio-init
pods on the minio NetworkPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All per-service NetworkPolicies used bare 'app: postgres/redis/minio'
selectors but infrastructure pods use 'app.kubernetes.io/name'. This
blocked all egress to databases, causing initContainer hangs and
service connectivity failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pod-level runAsNonRoot:true conflicts with initContainer runAsUser:0.
Kubernetes rejects the container. Add explicit override so the
initContainer can still run as root to fetch binaries from S3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab webservice/sidekiq share the foxhunt namespace but don't have
part-of=foxhunt label. Add release=gitlab selector to postgres and
redis NetworkPolicies so GitLab can reach its database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
default-deny and infrastructure ingress rules match correctly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh.
Also updated references in postgres.yaml, migrate.yaml,
and postgres-init.yaml to use db-credentials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Least-privilege ingress/egress for all 8 services + IB gateway +
infrastructure pods (postgres, redis, minio, questdb). Each policy
restricts which pods can communicate on which ports, limiting lateral
movement in the cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Establishes zero-trust network baseline for the foxhunt namespace:
- default-deny-all: blocks all ingress/egress unless explicitly allowed
- allow-dns: permits CoreDNS (kube-system) egress on UDP/TCP 53
- allow-monitoring-scrape: allows Prometheus ingress on metrics ports
(9091-9098) and Tempo OTLP gRPC egress on 4317
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training binaries export OTLP traces to Tempo when this env var is set.
Traces appear in foxhunt-traces Grafana dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
