From 638d7c733f84db8b2148fad10db312749d08f6ef Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 1 Mar 2026 22:25:11 +0100 Subject: [PATCH] docs: training observability + Grafana dashboards design GPU metrics polling, per-epoch RL metrics, OTLP tracing in training binaries, and three Grafana dashboards (training cockpit, traces, infra cockpit trace row). Co-Authored-By: Claude Opus 4.6 --- ...026-03-01-training-observability-design.md | 184 ++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 docs/plans/2026-03-01-training-observability-design.md diff --git a/docs/plans/2026-03-01-training-observability-design.md b/docs/plans/2026-03-01-training-observability-design.md new file mode 100644 index 000000000..6e4ccd8ed --- /dev/null +++ b/docs/plans/2026-03-01-training-observability-design.md @@ -0,0 +1,184 @@ +# Training Binary Observability + Grafana Dashboards Design + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Instrument training binaries with GPU metrics polling, per-epoch Prometheus metrics, and OTLP tracing so we can validate whether the GPU is maxed out during real DQN/PPO training runs. Deploy Grafana dashboards to visualize everything. + +**Architecture:** Training binaries poll `nvidia-smi` every 5s for GPU metrics, emit per-fold training metrics via existing Prometheus helpers, and export OTLP traces to Tempo. Grafana consumes Prometheus (metrics), Loki (logs), and Tempo (traces) with cross-linking. + +**Tech Stack:** Rust tracing + tracing-opentelemetry (already in workspace), Prometheus client (already used), nvidia-smi CLI, Grafana 8.x dashboards (JSON ConfigMaps). + +--- + +## Problem Statement + +The training binaries (`train_baseline_rl`, `train_baseline_supervised`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`) have a Prometheus metrics server running on port 9094 with 18 training metrics registered, but: + +1. **Zero GPU metrics emitted** — `nvidia-smi` is called once at startup for batch sizing, never polled during training +2. **`train_baseline_rl.rs` is a metrics black box** — DQN/PPO trainers are delegated to but never call back into `common::metrics::training_metrics`. Only `record_data_load()` and `set_active_workers()` are called +3. **No OTLP tracing** — training binaries use plain `tracing_subscriber::fmt()`, not `init_observability()` +4. **No `OTEL_EXPORTER_OTLP_ENDPOINT`** in K8s training job template +5. **No Grafana dashboards** for training or traces (design exists, implementation doesn't) + +## Data Flow + +``` +Training Binary → nvidia-smi poll (5s) → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana +Training Binary → set_epoch_loss() etc. → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana +Training Binary → init_observability() → OTLP gRPC :4317 → Tempo → Grafana (Traces) +Training Binary → stdout → containerd → Promtail → Loki → Grafana (linked via trace_id) +``` + +## Component 1: GPU Metrics Polling + +New file: `crates/common/src/metrics/gpu_metrics.rs` + +### Metrics Registered + +| Metric Name | Type | Description | +|---|---|---| +| `foxhunt_gpu_utilization_percent` | Gauge | SM utilization 0-100 | +| `foxhunt_gpu_memory_used_bytes` | Gauge | VRAM used in bytes | +| `foxhunt_gpu_memory_total_bytes` | Gauge | VRAM total in bytes | +| `foxhunt_gpu_temperature_celsius` | Gauge | GPU temperature | +| `foxhunt_gpu_power_draw_watts` | Gauge | Power consumption | + +### Implementation + +- `poll_gpu_metrics()` — shells out to `nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits`, parses CSV, sets gauges. MiB values from nvidia-smi are converted to bytes. Silently logs warning and skips update on parse failure (nvidia-smi not available on CPU-only machines). +- `start_gpu_poller(interval_secs: u64)` — spawns a daemon thread named `"gpu-metrics-poller"` that calls `poll_gpu_metrics()` in a loop with `std::thread::sleep(Duration::from_secs(interval_secs))`. Returns immediately. Thread is detached (no join handle needed). + +### Integration + +All 4 training binaries call `gpu_metrics::start_gpu_poller(5)` in `main()` right after `metrics_server::start_metrics_server(9094)`. + +## Component 2: Wire Per-Epoch Metrics into train_baseline_rl.rs + +Since DQN/PPO trainers return only a single `f64` (best validation loss), we instrument at the **fold level** in the binary, not inside the trainers. + +### Per-Fold Metrics Calls + +After each DQN/PPO fold completes: +```rust +metrics::set_epoch(model_name, fold_idx, fold_idx as f64); // fold as "epoch" +metrics::set_epoch_loss(model_name, fold_idx, best_val_loss); +metrics::set_validation_loss(model_name, fold_idx, best_val_loss); +metrics::set_iteration_seconds(model_name, fold_idx, fold_elapsed.as_secs_f64()); +``` + +After data loading: +```rust +metrics::record_data_load(model_name, load_elapsed.as_secs_f64()); // already done +``` + +On checkpoint save (in the checkpoint callback): +```rust +metrics::record_checkpoint_save(model_name, fold_idx, ckpt_bytes.len(), save_elapsed); +``` + +### Hyperopt Binaries + +`hyperopt_baseline_rl.rs` and `hyperopt_baseline_supervised.rs` — add `set_epoch()` and `set_epoch_loss()` per trial iteration. Lower priority since hyperopt is exploratory, but the metrics are cheap. + +## Component 3: OTLP Tracing in Training Binaries + +### Change + +In all 4 training binary `main()` functions, replace: +```rust +tracing_subscriber::fmt() + .with_env_filter(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"))) + .init(); +``` + +With: +```rust +common::observability::init_observability( + "train_baseline_rl", // service name varies per binary + std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok().as_deref(), +); +``` + +When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, `init_observability` falls back to console-only logging (same as today). When set, spans flow to Tempo. + +### K8s Training Job Template + +Add to `infra/k8s/training/job-template.yaml` pod env: +```yaml +- name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://tempo.foxhunt.svc.cluster.local:4317" +``` + +## Component 4: Grafana Training Cockpit Dashboard + +New file: `infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json` + +Provisioned as a ConfigMap, auto-loaded by Grafana's dashboard provisioner (same pattern as existing `foxhunt-cockpit.json`). + +### Dashboard Layout (8 rows) + +Per the existing design doc's Component 4 spec, using the actual metric names from `common/src/metrics/training_metrics.rs` and the new GPU metrics: + +| Row | Focus | Panels | +|---|---|---| +| 1 | Active Jobs | active_workers gauge, fold progress | +| 2 | Training Progress | epoch_loss + validation_loss timeseries, batches_per_second, iteration_seconds | +| 3 | GPU Resources | gpu_utilization_percent, gpu_memory_used/total, gpu_temperature, gpu_power_draw | +| 4 | Model Quality | eval_accuracy, eval_f1, eval_precision, eval_recall | +| 5 | Checkpoints | checkpoint_saves_total, checkpoint_failures_total, checkpoint_duration_seconds, checkpoint_size_bytes | +| 6 | Data Pipeline | batches_processed, data_load_seconds | +| 7 | Errors | nan_detected_total, gradient_explosion_total, feature_errors_total | +| 8 | Timing | iteration_seconds histogram, data_load_seconds histogram | + +### Template Variables + +- `$model` — dropdown: `dqn`, `ppo`, `tft`, etc. Filters all panels by `model` label +- `$fold` — dropdown: `0`, `1`, `2`, etc. Filters by `fold` label +- `$interval` — auto interval for rate() queries + +## Component 5: Grafana Traces Dashboard + +New file: `infra/k8s/monitoring/dashboards/foxhunt-traces.json` + +- Service map panel (Tempo node graph) +- Trace search table (service, operation, duration, status filters) +- Trace duration histogram +- Error rate by service timeseries + +## Component 6: Existing Dashboard Updates + +- `foxhunt-cockpit.json` — add one "Trace Health" row: spans/sec and error rate from Tempo metrics generator +- Service annotations — add Prometheus scrape annotations to 8 service deployment YAMLs (per existing design doc Component 1) + +## Component 7: Service OTLP Wiring + +Per existing design doc Component 2: Add `init_tracing()` call to 8 service `main()` functions. Add `#[instrument(skip_all)]` to ~20-30 key async functions (gRPC handlers, DB queries, ML inference paths). No instrumentation on hot-path tick processing. + +## File Changes Summary + +| Category | Files | Change | +|----------|-------|--------| +| New GPU metrics module | `common/src/metrics/gpu_metrics.rs` | nvidia-smi polling + 5 gauges | +| Training binary metrics | `ml/examples/train_baseline_rl.rs` | Wire per-fold metrics calls | +| Training binary OTLP | 4 training binary `main()` functions | Replace fmt with init_observability | +| K8s training job | `infra/k8s/training/job-template.yaml` | Add OTEL endpoint env var | +| New dashboards | 2 JSON files | training-cockpit, traces | +| Updated dashboards | 1 JSON file | cockpit (trace row) | +| K8s service annotations | 8 service YAMLs | Prometheus scrape annotations | +| Service OTLP | 8 service main.rs | Add init_tracing() | +| Instrumentation | ~20-30 key functions | `#[instrument(skip_all)]` | + +## What We Don't Change + +- No changes to DQN/PPO trainer internals (instrument at fold level in binaries) +- No changes to hot-path tick processing (no `#[instrument]` on SIMD/RDTSC paths) +- No new Rust dependencies (tracing-opentelemetry already in workspace) +- No changes to Tempo, Loki, or Prometheus deployment configs +- Existing infra cockpit layout stays as-is (just one row added) + +## Risks and Mitigations + +- **nvidia-smi subprocess overhead**: 5s poll interval, `nvidia-smi` takes ~50ms. 1% overhead on a 5s window. Acceptable. +- **nvidia-smi unavailable**: CPU-only dev machines don't have it. Poller logs a warning once and exits the thread gracefully. +- **Trace volume from training**: Training runs are ~30-60min with ~1000 folds. At one span per fold, volume is negligible (~1000 spans per job). +- **Metric cardinality**: `model` × `fold` labels. Max ~10 models × ~50 folds = 500 series. Low cardinality.