Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes for "no metrics" on the training dashboard:
1. Dashboard template variables ($model, $fold) sourced from
foxhunt_training_current_epoch which isn't emitted until the first
epoch completes. Switch to foxhunt_training_step which fires from
step 500 onward.
2. train.sh pod template missing Prometheus annotations
(prometheus.io/scrape, port, path). Also add the
app.kubernetes.io/component label to the eval manifest so
evaluation pods are discoverable too.
3. DQN and PPO trainers only called set_epoch() at the END of each
epoch. Move the call to the TOP of the epoch loop so the gauge
exists from the first training iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined
All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.
Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus was configured to scrape pods via prometheus.io/scrape
annotations, but all services and training jobs used gitlab.com/
prefix from legacy GitLab-managed Prometheus — causing Prometheus
to never discover any foxhunt pods. This resulted in stale/missing
metrics on the Grafana training dashboard.
12 files updated across services/, gpu-overlays/, training/, and
monitoring/ (node-exporter, dcgm-exporter cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add clamp_max() to PromQL queries and hard Y-axis limits to prevent
early-epoch degenerate values from blowing up panel scaling (Sharpe
showing 12 instead of 1.3, Profit Factor at 30k).
Run Summary: clamp_max on Best Val Loss (5), Sharpe (5), Profit Factor (20)
Training Quality: clamp_max on Epoch Sharpe (5), Sortino (10), PF (20)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stat panels with timeseries using smooth line interpolation,
gradient fill, and multi-tooltip. Layout changed from 6x1 to 3x2 grid.
Legends show model/fold only when multiple series exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 stat panels showing overall training run metrics: Best Val Loss,
Best Sharpe, Best Win Rate, Min Max Drawdown, Total Epochs, and Best
Profit Factor. Placed between Training Status and Training Curves rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add api_uptime_seconds gauge to API service (5s update interval),
matching the pattern used by trading/backtesting/ml-training services
- Remove obsolete Web Gateway Uptime and Active WebSocket Connections
panels from cockpit (web-gateway is not deployed)
- Add API Gateway Uptime (api_uptime_seconds) and API Auth Rate
(api_auth_requests_total) panels in their place
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Join node metrics with node_uname_info to resolve instance IPs to
hostnames (e.g. scw-foxhunt-platform-694db...) in all 10 node-level
panels: CPU, Memory, Disk I/O, Network I/O, Load Average, Disk Usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add __meta_kubernetes_pod_phase=Running filter to training-pods job.
Completed/failed Argo workflow step pods were showing as DOWN targets
because their containers are no longer running to serve /metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace api_gateway_ with api_ prefix to match actual metric exports (cockpit, observability)
- Change grpc_code!="OK" to grpc_code!="0" — gRPC uses numeric status codes (cockpit, trading)
- Update gRPC service filter to actual exported_service labels: trading-agent-service,
broker-gateway, backtesting-service, ml-training-service, data-acquisition-service (trading)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate annotated-pods and gitlab-annotated-pods scrape jobs
(all targets already covered by dedicated jobs)
- Drop Argo controller from foxhunt-services job (dedicated HTTPS job exists)
- Resolve datasource variable placeholders to actual UIDs in dashboard JSON
- Add Ingress rule to Prometheus netpol allowing Grafana on port 9090
Targets: 49 → 18 (zero duplicates)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Infrastructure Cockpit:
- GPU gauges → side-by-side timeseries (GPU Utilization + GPU VRAM with total line)
- Added Memory by Service timeseries next to existing CPU by Service
- Cluster resource gauges shrunk to h=4 for tighter layout
Training Cockpit:
- GPU Utilization/Memory gauges → side-by-side timeseries matching infra cockpit
- GPU Temperature/Power widened from w=6 to w=12 in paired row
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 stat panels (Ensemble Agreement, Prediction Confidence, Model Accuracy,
Ensemble Sharpe, Realized PnL, Unrealized PnL) showed red when ML metrics
don't exist yet. Added special null mapping → "Inactive" in dark-blue.
Panels will still show red/yellow/green when the trading system is active
and emitting metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Active CI Jobs" stat panel used sum(gitlab_ci_active_jobs_sum) which
is a histogram accumulator that only grows (was showing 982 red). Changed
to sum(increase(pipeline_processing_events_total[1h])) which shows actual
pipeline activity in the last hour. Threshold changed to blue=idle,
green=active.
Also fixed Grafana provisioning:
- Removed overlapping named providers (cockpits/operations/infrastructure/cicd)
that duplicated sidecarProvider, causing UID conflicts and write lockout
- Enabled foldersFromFilesStructure on sidecarProvider
- Unlabeled grafana-dashboards-foxhunt ConfigMap (duplicate of cockpits+operations)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New row shows GitLab, CI Runners, Monitoring, Collectors, Tailscale,
and Platform Total stat panels, plus a pod status table, CPU/memory
timeseries, and container restart chart for all non-foxhunt pods.
Also fix minio-init-buckets job: remove part-of:foxhunt label (triggers
default-deny egress block) and rely on minio NetworkPolicy ingress
allowlist for minio-init pods instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes for GPU metrics not appearing in dashboards:
1. Added runtimeClassName: nvidia so DCGM can access NVML/GPU devices
2. Custom counters ConfigMap without DCP/profiling metrics (avoids
SYS_ADMIN requirement that caused fatal crash)
3. Added gitlab.com/prometheus_* annotations so GitLab's Prometheus
discovers and scrapes the DCGM pods
Increased memory limit 128Mi → 1Gi (was OOMKilled).
Metrics now flowing: gpu_utilization, fb_used/free, power_usage, temps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The L4 pool was unused: all training routes to ci-training (L40S) and
all compilation routes to ci-compile-cpu (POP2). Disabled in terragrunt
and applied to destroy the pool. Removed all ci-rl references from K8s
manifests and CI comments.
Final pool layout:
- ci-compile-cpu (POP2-32C-128G) — Rust compilation
- ci-training (L40S-1-48G) — all GPU training + hyperopt
- services (DEV1-L) — production services
- gitlab (GP1-XS) — GitLab CE
- gpu-dev (GP1-L) — DevPod development
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Applied terragrunt to recreate the L4 GPU pool with the correct name
`ci-rl` (was `ci-compile` due to immutable Scaleway pool names).
Updated all K8s manifests and comments to match. Removed the 3 stale
`moved` blocks from main.tf since the state renames are now applied.
Pool naming is now consistent across Terraform, Scaleway, and K8s configs:
- ci-compile-cpu (POP2-32C-128G) — CPU compilation
- ci-rl (L4-1-24G) — RL training / CUDA compile
- ci-training (L40S-1-48G) — supervised training + hyperopt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool mapping:
- ci-compile-cpu (POP2-32C-128G): Rust compile, web dashboard, manifest
- ci-compile (L4): CUDA compile with stubs only
- ci-training (L40S): ALL training (RL + supervised)
Main runner default → ci-compile-cpu
RL runner default → ci-training
.train-rl-base → explicit ci-training node selector
Replaced all stale ci-rl references with correct pool names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo was OOMKilled every ~30 min at 512Mi, causing web-gateway OTLP
export errors. Increased to 1Gi.
RL runner had cpu_request_overwrite_max_allowed=4000m but hyperopt RL
jobs need 6000m for parallel PSO trials. Increased to 8000m.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab's bundled Prometheus only scrapes pods with gitlab.com/prometheus_scrape
annotations, not the standard prometheus.io/* ones. Without this, the
Infrastructure Cockpit shows no data for node/pod/cluster metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
