Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New row shows GitLab, CI Runners, Monitoring, Collectors, Tailscale,
and Platform Total stat panels, plus a pod status table, CPU/memory
timeseries, and container restart chart for all non-foxhunt pods.
Also fix minio-init-buckets job: remove part-of:foxhunt label (triggers
default-deny egress block) and rely on minio NetworkPolicy ingress
allowlist for minio-init pods instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The minio-init-buckets job pod has label app.kubernetes.io/name: minio-init
which doesn't match part-of: foxhunt (adding part-of would trigger
default-deny egress). Add explicit ingress allowlist entry for minio-init
pods on the minio NetworkPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All per-service NetworkPolicies used bare 'app: postgres/redis/minio'
selectors but infrastructure pods use 'app.kubernetes.io/name'. This
blocked all egress to databases, causing initContainer hangs and
service connectivity failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pod-level runAsNonRoot:true conflicts with initContainer runAsUser:0.
Kubernetes rejects the container. Add explicit override so the
initContainer can still run as root to fetch binaries from S3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab webservice/sidekiq share the foxhunt namespace but don't have
part-of=foxhunt label. Add release=gitlab selector to postgres and
redis NetworkPolicies so GitLab can reach its database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
default-deny and infrastructure ingress rules match correctly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh.
Also updated references in postgres.yaml, migrate.yaml,
and postgres-init.yaml to use db-credentials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Least-privilege ingress/egress for all 8 services + IB gateway +
infrastructure pods (postgres, redis, minio, questdb). Each policy
restricts which pods can communicate on which ports, limiting lateral
movement in the cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Establishes zero-trust network baseline for the foxhunt namespace:
- default-deny-all: blocks all ingress/egress unless explicitly allowed
- allow-dns: permits CoreDNS (kube-system) egress on UDP/TCP 53
- allow-monitoring-scrape: allows Prometheus ingress on metrics ports
(9091-9098) and Tempo OTLP gRPC egress on 4317
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training binaries export OTLP traces to Tempo when this env var is set.
Traces appear in foxhunt-traces Grafana dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL runner only handles GPU training jobs, so global nvidia runtime
is safe and required (KUBERNETES_RUNTIME_CLASS_NAME override wasn't
being applied). Main runner keeps overwrite-allowed pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove runtime_class_name="nvidia" from runner configs (applied to ALL
pods, broke compile on CPU-only ci-compile-cpu nodes). Instead use
runtime_class_name_overwrite_allowed + KUBERNETES_RUNTIME_CLASS_NAME
per-job on GPU templates (.train-rl-base, .train-validate-base).
Also routes build-web-dashboard and write-manifest to ci-compile-cpu.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cert-manager stores certs as tls.crt/tls.key but MinIO expects
public.crt/private.key — add volume items projection.
mc client doesn't use MC_CA_CERTIFICATE env; instead copy CA cert to
~/.mc/certs/CAs/ before running mc alias set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch all MinIO communication from HTTP to HTTPS across the entire stack:
- MinIO deployment: mount TLS secret, tcpSocket probes, HTTPS init-buckets
- 11 service YAMLs: HTTPS rclone endpoint + CA cert volume mount
- Training job template + train.sh: HTTPS for fetch-binaries and uploader
- CI pipeline (.gitlab-ci.yml): all 7 rclone exports use HTTPS + CA cert
- GitLab runner values: minio-ca-cert ConfigMap volume for CI pods
- Rust: CA cert loading via MINIO_CA_CERT_PATH in training_uploader,
storage backend, data_acquisition uploader, and k8s_dispatcher
- Cert generation script (infra/scripts/generate-minio-tls.sh)
Fixes training uploader S3 upload failures caused by with_allow_http(false)
connecting to an HTTP endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The job template referenced a non-existent 'gpu-training' pool. The
actual L40S GPU pool in Kapsule is named 'ci-training' (scale-from-zero,
0→1 autoscaling).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The existing `global.appConfig.gitlab_kas.enabled: false` only affected
the KAS subchart config but was ignored by the chart template that
generates gitlab.yml. Adding `global.kas.enabled: false` properly sets
`gitlab_kas.enabled: false` in gitlab.yml and removes the orphaned
gitlab-kas Service, stopping ~5 GRPC::Unavailable errors per git push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder metric names (traces_spanmetrics_calls_total,
traces_spanmetrics_latency_*) with real Tempo metrics
(tempo_distributor_spans_received_total, tempo_spanmetrics_latency_*).
Switch Recent Traces panel from nativeSearch to TraceQL query.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all placeholder PromQL expressions with real metric names
that match actual Prometheus metrics emitted by the trading system.
Add $model, $symbol, and $service template variables for filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traces_spanmetrics_calls_total with tempo_distributor_spans_received_total
in all 3 Trace Health panels (Spans/sec, Trace Error Rate, Spans/sec by Service).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs found during validation:
1. rclone defaults to HTTPS when endpoint has no scheme — MinIO serves
plain HTTP. All RCLONE_S3_ENDPOINT values now include http:// prefix.
2. rclone copyto returns exit 0 for non-existent S3 keys (empty bucket).
initContainers crashed on chmod of missing file instead of falling
through to PVC cache. Added `&& [ -f /binaries/$SERVICE ]` guard.
Also: deploy stage uses `kubectl apply -k` (not -f) for minio/ dir
to properly handle kustomization.yaml.
Deleted stale s3-credentials secret from cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all compiled binary uploads/downloads from public Scaleway S3 to
MinIO running inside Kapsule. Keeps proprietary binaries and trained
models off the public internet.
- CI compile/upload jobs: Scaleway → MinIO endpoint + credentials
- train.sh: all 3 rclone blocks → RCLONE_CONFIG_MINIO_*
- Deploy stage: apply MinIO manifests + wait for readiness before rollout
- Add infra/k8s/minio/ (Deployment, Service, Secret, PVC, init-buckets Job)
- Update s3-credentials.secret.example → minio-credentials format
IaC Terragrunt jobs and SCW registry pulls intentionally unchanged.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- foxhunt-trading-cockpit.json: 22 panels covering signal quality,
execution, risk, ML model health, and latency
- foxhunt-training-cockpit.json: 36 panels covering active jobs,
training progress, GPU resources, model quality, checkpoints,
data pipeline, errors, and job history
- foxhunt-traces.json: 12 panels with service map, trace search,
latency by service, and error rate from Tempo
- foxhunt-cockpit.json: added Trace Health row (spans/sec, error rate,
spans by service) and DS_TEMPO datasource input
- import.sh: added 3 new dashboards to ConfigMap group and API fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable GitLab Prometheus to auto-discover and scrape metrics from all
foxhunt pods by adding gitlab.com/prometheus_scrape annotations to the
pod template metadata of 9 service deployments and the training job
template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web-gateway routing:
- Point TRADING_SERVICE_URL at api-gateway (proto mismatch fix)
Web-gateway uses foxhunt.tli.TradingService proto but was connecting
directly to trading-service which implements trading.TradingService.
api-gateway already proxies Subscribe* → Stream* correctly.
GitLab KAS:
- Disable gitlab_kas in appConfig to stop sidekiq NotifyGitPushWorker
errors (KAS pod was already disabled but Rails still tried to connect)
Trading service monitoring (3 stubs → real):
- AcknowledgeAlert: real alert lookup + state mutation in shared store
- GetActiveAlerts: returns actual active alerts from in-memory store
- StreamAlerts: now persists generated alerts (capped at 1000 entries)
Trading service ML streams (2 stubs → real):
- StreamModelMetrics: emits real inference_count, error_count, latency
per model every N seconds from the RuntimeModelInfo registry
- StreamSignalStrength: emits per-symbol signal aggregation from model
ensemble weights and latency confidence
Backtesting service:
- stop_backtest: real CancellationToken cancellation (was no-op)
Tokens stored per-backtest, execute_backtest wraps strategy call
in tokio::select! for immediate cancellation
Deleted 7 empty placeholder files:
- 4 Wave D regime stubs (dynamic_stops, ensemble, performance_tracker,
position_sizer) — comment-only files, never wired
- 2 Wave 3 feature stubs (microstructure, statistical)
- 1 PPO stub (unified_ppo.rs — empty struct definitions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ohlcv_bars table migration (050) for trading_service prediction loop
- Fix kube-state-metrics RBAC: add admissionregistration.k8s.io and
certificates.k8s.io API groups to suppress forbidden errors
- Fix web-gateway gRPC stream reconnect: detect Unimplemented via both
status code and error message text (proxy may remap codes)
- Add automated migration step to CI deploy pipeline with tracking table
(_applied_migrations) so migrations are applied before service rollouts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana dashboards are provisioned via sidecar ConfigMaps (label
grafana_dashboard=1), making the API endpoint reject overwrites. The
import script now resolves ${DS_*} variables and deploys via kubectl
ConfigMap updates with automatic Grafana restart. Falls back to API
import when kubectl is unavailable or SKIP_CONFIGMAP=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add foxhunt-cockpit.json — single-page overview with 7 rows (health bar,
service status, resources, GPU/training, restarts/network, errors, live
logs) pulling critical signals from all domain dashboards.
Migrate all 10 dashboards from hardcoded datasource UIDs to ${DS_PROMETHEUS},
${DS_LOKI}, ${DS_TEMPO} variables with __inputs sections for portable import.
Update import.sh to auto-detect datasource UIDs from Grafana API at deploy
time, using /api/dashboards/import endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
