- Pure-Python scraper replaces shell+Python hybrid (fixes pushgateway
RemoteDisconnected crash caused by duplicate volume name labels)
- Scraper dynamically discovers all Scaleway resources via API:
billing, instances, K8s pools, block volumes, IPs
- 40-panel cockpit dashboard with fully dynamic tables (auto-adapts
when infra changes — no hardcoded instance/volume names)
- Volume table keyed by UUID (not truncated name) to prevent collisions
- Label sanitization for Prometheus text format safety
- Pushgateway push via PUT + Content-Type: text/plain; version=0.0.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
74-panel dashboard for in-depth pipeline monitoring. Select a running pod
to see training curves, GPU health (DCGM), eval metrics, CI/Argo workflows,
container resources, and live logs in one place. Stored in Postgres via API
(no ConfigMap restarts needed).
Sections: Job Overview, CI Pipeline & Argo Workflows, CI Logs, Training
Curves, Trading Performance, Evaluation Metrics, GPU & Hardware, Throughput,
Hyperopt Trials, Container Resources, Live Logs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new Grafana dashboards powered by Loki:
- CI Pipeline Logs: test gate, clippy, Redis sidecar, build/deploy, errors
- Training Pipeline Logs: epoch/loss, hyperopt trials, GPU/CUDA, walk-forward, data loading
Both use Promtail-extracted labels (level, container, pod) for efficient
stream selection. Collapsible sections keep overview clean while providing
deep drill-down. Variables for pipeline/job type, log level, and text search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all 3 cockpit dashboards (Cockpit, Trading Cockpit, Training
Cockpit) into a single "Cockpits" folder. GPU & Training Jobs moved
to Infrastructure. 4 folders total: Cockpits, Operations,
Infrastructure, CI-CD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip "Foxhunt - " prefix from all dashboard titles since the folder
structure already provides context. Use clean folder names without
slashes (Trading, Training, Operations, Infrastructure, CI-CD) and
per-folder provisioning providers to prevent duplicate folder creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces single "Foxhunt" folder with Trading, Training, Operations,
Infrastructure, and CI/CD groupings. Updates ConfigMap definitions in
import.sh and Grafana Helm values to use foldersFromFilesStructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana sidecar supports grafana_folder annotations to place
dashboards in specific folders. Without this, all provisioned
dashboards land in the General folder. Add Foxhunt/Infrastructure/CI-CD
folder mapping to both CI pipeline and import.sh.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- foxhunt-trading-cockpit.json: 22 panels covering signal quality,
execution, risk, ML model health, and latency
- foxhunt-training-cockpit.json: 36 panels covering active jobs,
training progress, GPU resources, model quality, checkpoints,
data pipeline, errors, and job history
- foxhunt-traces.json: 12 panels with service map, trace search,
latency by service, and error rate from Tempo
- foxhunt-cockpit.json: added Trace Health row (spans/sec, error rate,
spans by service) and DS_TEMPO datasource input
- import.sh: added 3 new dashboards to ConfigMap group and API fallback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana dashboards are provisioned via sidecar ConfigMaps (label
grafana_dashboard=1), making the API endpoint reject overwrites. The
import script now resolves ${DS_*} variables and deploys via kubectl
ConfigMap updates with automatic Grafana restart. Falls back to API
import when kubectl is unavailable or SKIP_CONFIGMAP=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add foxhunt-cockpit.json — single-page overview with 7 rows (health bar,
service status, resources, GPU/training, restarts/network, errors, live
logs) pulling critical signals from all domain dashboards.
Migrate all 10 dashboards from hardcoded datasource UIDs to ${DS_PROMETHEUS},
${DS_LOKI}, ${DS_TEMPO} variables with __inputs sections for portable import.
Update import.sh to auto-detect datasource UIDs from Grafana API at deploy
time, using /api/dashboards/import endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
