Update all dashboard JSON files to use Helm chart's Prometheus datasource
UID and add Grafana sidecar folder annotations for auto-provisioning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three root causes for "no metrics" on the training dashboard:
1. Dashboard template variables ($model, $fold) sourced from
foxhunt_training_current_epoch which isn't emitted until the first
epoch completes. Switch to foxhunt_training_step which fires from
step 500 onward.
2. train.sh pod template missing Prometheus annotations
(prometheus.io/scrape, port, path). Also add the
app.kubernetes.io/component label to the eval manifest so
evaluation pods are discoverable too.
3. DQN and PPO trainers only called set_epoch() at the END of each
epoch. Move the call to the TOP of the epoch loop so the gauge
exists from the first training iteration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove phantom foxhunt_training_total_epochs reference (never registered)
and replace with foxhunt_training_step from the recent metrics commit.
Add full coverage for all 52 Tier-1 Prometheus metrics across 5 new rows:
RL diagnostics (Q-overestimation, PPO entropy/KL/advantage), training
health (NaN/grad explosion/feature errors, checkpoint ops, data load
latency), epoch returns, supervised eval (accuracy/precision/recall/F1),
and hyperopt details (mode, trial epoch, failed trials).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `source` template variable to the training dashboard that controls
which Prometheus job labels are queried:
- Live: only `training-pods` (running pods scraped directly)
- CI History: only `.*_baseline.*` (completed CI runs via pushgateway)
- All: both sources combined
All 37 panel queries now use `job=~"$source"` for consistent filtering.
Active Workers panel stays hardcoded to `job="training-pods"` since it
only makes sense for live pods. Template variable queries (model/fold
dropdowns) also respect the source selector.
Default is "Live" — dashboard shows only the currently running training
session with no pushgateway stale data contamination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stat panels (Active Workers, Current Epoch, Trial Progress, Elapsed, Best
Objective) showed conflicting values from pushgateway stale gauges overlapping
with live training pod metrics. Scoped all stat/gauge panels and template
variable queries to job="training-pods" so the dashboard reflects only the
currently running training session. Time-series panels remain unfiltered via
$model template variable scoping — dropdown only lists live models, so
historical pushgateway data is naturally excluded without explicit filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove cuda from default features in ml-core, ml-dqn, ml-ppo, ml
- Propagate cuda feature from ml → ml-core/ml-dqn/ml-ppo
- CI compile-training already uses --features ml/cuda explicitly
- Fix MaxDD log format: {:.1}% → {:.3}% (was rounding 0.033% to 0.0%)
- Suppress unused_labels/unused_variables warnings for cfg(cuda) code
- Add CALLBACK_ENDPOINT env to ml-training-service deployment
- Fix Grafana active_workers query to use sum() with fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add clamp_max() to PromQL queries and hard Y-axis limits to prevent
early-epoch degenerate values from blowing up panel scaling (Sharpe
showing 12 instead of 1.3, Profit Factor at 30k).
Run Summary: clamp_max on Best Val Loss (5), Sharpe (5), Profit Factor (20)
Training Quality: clamp_max on Epoch Sharpe (5), Sortino (10), PF (20)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stat panels with timeseries using smooth line interpolation,
gradient fill, and multi-tooltip. Layout changed from 6x1 to 3x2 grid.
Legends show model/fold only when multiple series exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 stat panels showing overall training run metrics: Best Val Loss,
Best Sharpe, Best Win Rate, Min Max Drawdown, Total Epochs, and Best
Profit Factor. Placed between Training Status and Training Curves rows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add api_uptime_seconds gauge to API service (5s update interval),
matching the pattern used by trading/backtesting/ml-training services
- Remove obsolete Web Gateway Uptime and Active WebSocket Connections
panels from cockpit (web-gateway is not deployed)
- Add API Gateway Uptime (api_uptime_seconds) and API Auth Rate
(api_auth_requests_total) panels in their place
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Join node metrics with node_uname_info to resolve instance IPs to
hostnames (e.g. scw-foxhunt-platform-694db...) in all 10 node-level
panels: CPU, Memory, Disk I/O, Network I/O, Load Average, Disk Usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace api_gateway_ with api_ prefix to match actual metric exports (cockpit, observability)
- Change grpc_code!="OK" to grpc_code!="0" — gRPC uses numeric status codes (cockpit, trading)
- Update gRPC service filter to actual exported_service labels: trading-agent-service,
broker-gateway, backtesting-service, ml-training-service, data-acquisition-service (trading)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate annotated-pods and gitlab-annotated-pods scrape jobs
(all targets already covered by dedicated jobs)
- Drop Argo controller from foxhunt-services job (dedicated HTTPS job exists)
- Resolve datasource variable placeholders to actual UIDs in dashboard JSON
- Add Ingress rule to Prometheus netpol allowing Grafana on port 9090
Targets: 49 → 18 (zero duplicates)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Infrastructure Cockpit:
- GPU gauges → side-by-side timeseries (GPU Utilization + GPU VRAM with total line)
- Added Memory by Service timeseries next to existing CPU by Service
- Cluster resource gauges shrunk to h=4 for tighter layout
Training Cockpit:
- GPU Utilization/Memory gauges → side-by-side timeseries matching infra cockpit
- GPU Temperature/Power widened from w=6 to w=12 in paired row
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 stat panels (Ensemble Agreement, Prediction Confidence, Model Accuracy,
Ensemble Sharpe, Realized PnL, Unrealized PnL) showed red when ML metrics
don't exist yet. Added special null mapping → "Inactive" in dark-blue.
Panels will still show red/yellow/green when the trading system is active
and emitting metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Active CI Jobs" stat panel used sum(gitlab_ci_active_jobs_sum) which
is a histogram accumulator that only grows (was showing 982 red). Changed
to sum(increase(pipeline_processing_events_total[1h])) which shows actual
pipeline activity in the last hour. Threshold changed to blue=idle,
green=active.
Also fixed Grafana provisioning:
- Removed overlapping named providers (cockpits/operations/infrastructure/cicd)
that duplicated sidecarProvider, causing UID conflicts and write lockout
- Enabled foldersFromFilesStructure on sidecarProvider
- Unlabeled grafana-dashboards-foxhunt ConfigMap (duplicate of cockpits+operations)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New row shows GitLab, CI Runners, Monitoring, Collectors, Tailscale,
and Platform Total stat panels, plus a pod status table, CPU/memory
timeseries, and container restart chart for all non-foxhunt pods.
Also fix minio-init-buckets job: remove part-of:foxhunt label (triggers
default-deny egress block) and rely on minio NetworkPolicy ingress
allowlist for minio-init pods instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter v3.x emits lowercase metric names (dcgm_gpu_utilization)
but dashboards used the old uppercase format (DCGM_FI_DEV_GPU_UTIL).
Fixed across 4 dashboard files:
- foxhunt-cockpit.json (Infrastructure Cockpit)
- foxhunt-training-cockpit.json (Training Cockpit)
- foxhunt-gpu-training.json (GPU Training)
- gpu-overview.json (GPU Overview)
Replaced DCGM_FI_DEV_FB_TOTAL with (dcgm_fb_used + dcgm_fb_free)
since DCGM v3 doesn't emit a total metric.
Redeployed all dashboard ConfigMaps and restarted Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dashboard Loki queries used naive `|= "ERROR"` string matching which
created a self-referential feedback loop: Loki and Grafana log query
execution text containing "ERROR" at INFO level, inflating the error
count by ~19/min with phantom matches.
Add `container!~"loki|grafana|grafana-sc-dashboard|prometheus-server"`
filter to all 11 affected queries across 4 dashboards (cockpit,
overview, logs, services).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 28 placeholder PromQL expressions with real metric names:
- Active Jobs: kube_job_status_active from kube-state-metrics
- Training Progress: foxhunt_training_epoch_loss, current_epoch, etc.
- GPU Resources: DCGM_FI_DEV_GPU_UTIL/FB_USED/GPU_TEMP/POWER_USAGE
- Model Quality: foxhunt_training_eval_accuracy/f1/precision/recall
- Checkpoints: foxhunt_training_checkpoint_saves/failures/duration/size
- Data Pipeline: foxhunt_training_batches_processed, data_load_seconds
- Errors: foxhunt_training_nan_detected/gradient_explosion/feature_errors
- Job History: kube_job_complete_time/start_time for duration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder metric names (traces_spanmetrics_calls_total,
traces_spanmetrics_latency_*) with real Tempo metrics
(tempo_distributor_spans_received_total, tempo_spanmetrics_latency_*).
Switch Recent Traces panel from nativeSearch to TraceQL query.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all placeholder PromQL expressions with real metric names
that match actual Prometheus metrics emitted by the trading system.
Add $model, $symbol, and $service template variables for filtering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace traces_spanmetrics_calls_total with tempo_distributor_spans_received_total
in all 3 Trace Health panels (Spans/sec, Trace Error Rate, Spans/sec by Service).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- foxhunt-trading-cockpit.json: 22 panels covering signal quality,
execution, risk, ML model health, and latency
- foxhunt-training-cockpit.json: 36 panels covering active jobs,
training progress, GPU resources, model quality, checkpoints,
data pipeline, errors, and job history
- foxhunt-traces.json: 12 panels with service map, trace search,
latency by service, and error rate from Tempo
- foxhunt-cockpit.json: added Trace Health row (spans/sec, error rate,
spans by service) and DS_TEMPO datasource input
- import.sh: added 3 new dashboards to ConfigMap group and API fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana dashboards are provisioned via sidecar ConfigMaps (label
grafana_dashboard=1), making the API endpoint reject overwrites. The
import script now resolves ${DS_*} variables and deploys via kubectl
ConfigMap updates with automatic Grafana restart. Falls back to API
import when kubectl is unavailable or SKIP_CONFIGMAP=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add foxhunt-cockpit.json — single-page overview with 7 rows (health bar,
service status, resources, GPU/training, restarts/network, errors, live
logs) pulling critical signals from all domain dashboards.
Migrate all 10 dashboards from hardcoded datasource UIDs to ${DS_PROMETHEUS},
${DS_LOKI}, ${DS_TEMPO} variables with __inputs sections for portable import.
Update import.sh to auto-detect datasource UIDs from Grafana API at deploy
time, using /api/dashboards/import endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
