docs: training observability + Grafana dashboards design

GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/plans/2026-03-01-training-observability-design.md

@@ -0,0 +1,184 @@
+# Training Binary Observability + Grafana Dashboards Design
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Instrument training binaries with GPU metrics polling, per-epoch Prometheus metrics, and OTLP tracing so we can validate whether the GPU is maxed out during real DQN/PPO training runs. Deploy Grafana dashboards to visualize everything.
+
+**Architecture:** Training binaries poll `nvidia-smi` every 5s for GPU metrics, emit per-fold training metrics via existing Prometheus helpers, and export OTLP traces to Tempo. Grafana consumes Prometheus (metrics), Loki (logs), and Tempo (traces) with cross-linking.
+
+**Tech Stack:** Rust tracing + tracing-opentelemetry (already in workspace), Prometheus client (already used), nvidia-smi CLI, Grafana 8.x dashboards (JSON ConfigMaps).
+
+---
+
+## Problem Statement
+
+The training binaries (`train_baseline_rl`, `train_baseline_supervised`, `hyperopt_baseline_rl`, `hyperopt_baseline_supervised`) have a Prometheus metrics server running on port 9094 with 18 training metrics registered, but:
+
+1. **Zero GPU metrics emitted** — `nvidia-smi` is called once at startup for batch sizing, never polled during training
+2. **`train_baseline_rl.rs` is a metrics black box** — DQN/PPO trainers are delegated to but never call back into `common::metrics::training_metrics`. Only `record_data_load()` and `set_active_workers()` are called
+3. **No OTLP tracing** — training binaries use plain `tracing_subscriber::fmt()`, not `init_observability()`
+4. **No `OTEL_EXPORTER_OTLP_ENDPOINT`** in K8s training job template
+5. **No Grafana dashboards** for training or traces (design exists, implementation doesn't)
+
+## Data Flow
+
+```
+Training Binary → nvidia-smi poll (5s) → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana
+Training Binary → set_epoch_loss() etc. → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana
+Training Binary → init_observability() → OTLP gRPC :4317 → Tempo → Grafana (Traces)
+Training Binary → stdout → containerd → Promtail → Loki → Grafana (linked via trace_id)
+```
+
+## Component 1: GPU Metrics Polling
+
+New file: `crates/common/src/metrics/gpu_metrics.rs`
+
+### Metrics Registered
+
+| Metric Name | Type | Description |
+|---|---|---|
+| `foxhunt_gpu_utilization_percent` | Gauge | SM utilization 0-100 |
+| `foxhunt_gpu_memory_used_bytes` | Gauge | VRAM used in bytes |
+| `foxhunt_gpu_memory_total_bytes` | Gauge | VRAM total in bytes |
+| `foxhunt_gpu_temperature_celsius` | Gauge | GPU temperature |
+| `foxhunt_gpu_power_draw_watts` | Gauge | Power consumption |
+
+### Implementation
+
+- `poll_gpu_metrics()` — shells out to `nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits`, parses CSV, sets gauges. MiB values from nvidia-smi are converted to bytes. Silently logs warning and skips update on parse failure (nvidia-smi not available on CPU-only machines).
+- `start_gpu_poller(interval_secs: u64)` — spawns a daemon thread named `"gpu-metrics-poller"` that calls `poll_gpu_metrics()` in a loop with `std::thread::sleep(Duration::from_secs(interval_secs))`. Returns immediately. Thread is detached (no join handle needed).
+
+### Integration
+
+All 4 training binaries call `gpu_metrics::start_gpu_poller(5)` in `main()` right after `metrics_server::start_metrics_server(9094)`.
+
+## Component 2: Wire Per-Epoch Metrics into train_baseline_rl.rs
+
+Since DQN/PPO trainers return only a single `f64` (best validation loss), we instrument at the **fold level** in the binary, not inside the trainers.
+
+### Per-Fold Metrics Calls
+
+After each DQN/PPO fold completes:
+```rust
+metrics::set_epoch(model_name, fold_idx, fold_idx as f64);  // fold as "epoch"
+metrics::set_epoch_loss(model_name, fold_idx, best_val_loss);
+metrics::set_validation_loss(model_name, fold_idx, best_val_loss);
+metrics::set_iteration_seconds(model_name, fold_idx, fold_elapsed.as_secs_f64());
+```
+
+After data loading:
+```rust
+metrics::record_data_load(model_name, load_elapsed.as_secs_f64());  // already done
+```
+
+On checkpoint save (in the checkpoint callback):
+```rust
+metrics::record_checkpoint_save(model_name, fold_idx, ckpt_bytes.len(), save_elapsed);
+```
+
+### Hyperopt Binaries
+
+`hyperopt_baseline_rl.rs` and `hyperopt_baseline_supervised.rs` — add `set_epoch()` and `set_epoch_loss()` per trial iteration. Lower priority since hyperopt is exploratory, but the metrics are cheap.
+
+## Component 3: OTLP Tracing in Training Binaries
+
+### Change
+
+In all 4 training binary `main()` functions, replace:
+```rust
+tracing_subscriber::fmt()
+    .with_env_filter(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
+    .init();
+```
+
+With:
+```rust
+common::observability::init_observability(
+    "train_baseline_rl",  // service name varies per binary
+    std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok().as_deref(),
+);
+```
+
+When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, `init_observability` falls back to console-only logging (same as today). When set, spans flow to Tempo.
+
+### K8s Training Job Template
+
+Add to `infra/k8s/training/job-template.yaml` pod env:
+```yaml
+- name: OTEL_EXPORTER_OTLP_ENDPOINT
+  value: "http://tempo.foxhunt.svc.cluster.local:4317"
+```
+
+## Component 4: Grafana Training Cockpit Dashboard
+
+New file: `infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json`
+
+Provisioned as a ConfigMap, auto-loaded by Grafana's dashboard provisioner (same pattern as existing `foxhunt-cockpit.json`).
+
+### Dashboard Layout (8 rows)
+
+Per the existing design doc's Component 4 spec, using the actual metric names from `common/src/metrics/training_metrics.rs` and the new GPU metrics:
+
+| Row | Focus | Panels |
+|---|---|---|
+| 1 | Active Jobs | active_workers gauge, fold progress |
+| 2 | Training Progress | epoch_loss + validation_loss timeseries, batches_per_second, iteration_seconds |
+| 3 | GPU Resources | gpu_utilization_percent, gpu_memory_used/total, gpu_temperature, gpu_power_draw |
+| 4 | Model Quality | eval_accuracy, eval_f1, eval_precision, eval_recall |
+| 5 | Checkpoints | checkpoint_saves_total, checkpoint_failures_total, checkpoint_duration_seconds, checkpoint_size_bytes |
+| 6 | Data Pipeline | batches_processed, data_load_seconds |
+| 7 | Errors | nan_detected_total, gradient_explosion_total, feature_errors_total |
+| 8 | Timing | iteration_seconds histogram, data_load_seconds histogram |
+
+### Template Variables
+
+- `$model` — dropdown: `dqn`, `ppo`, `tft`, etc. Filters all panels by `model` label
+- `$fold` — dropdown: `0`, `1`, `2`, etc. Filters by `fold` label
+- `$interval` — auto interval for rate() queries
+
+## Component 5: Grafana Traces Dashboard
+
+New file: `infra/k8s/monitoring/dashboards/foxhunt-traces.json`
+
+- Service map panel (Tempo node graph)
+- Trace search table (service, operation, duration, status filters)
+- Trace duration histogram
+- Error rate by service timeseries
+
+## Component 6: Existing Dashboard Updates
+
+- `foxhunt-cockpit.json` — add one "Trace Health" row: spans/sec and error rate from Tempo metrics generator
+- Service annotations — add Prometheus scrape annotations to 8 service deployment YAMLs (per existing design doc Component 1)
+
+## Component 7: Service OTLP Wiring
+
+Per existing design doc Component 2: Add `init_tracing()` call to 8 service `main()` functions. Add `#[instrument(skip_all)]` to ~20-30 key async functions (gRPC handlers, DB queries, ML inference paths). No instrumentation on hot-path tick processing.
+
+## File Changes Summary
+
+| Category | Files | Change |
+|----------|-------|--------|
+| New GPU metrics module | `common/src/metrics/gpu_metrics.rs` | nvidia-smi polling + 5 gauges |
+| Training binary metrics | `ml/examples/train_baseline_rl.rs` | Wire per-fold metrics calls |
+| Training binary OTLP | 4 training binary `main()` functions | Replace fmt with init_observability |
+| K8s training job | `infra/k8s/training/job-template.yaml` | Add OTEL endpoint env var |
+| New dashboards | 2 JSON files | training-cockpit, traces |
+| Updated dashboards | 1 JSON file | cockpit (trace row) |
+| K8s service annotations | 8 service YAMLs | Prometheus scrape annotations |
+| Service OTLP | 8 service main.rs | Add init_tracing() |
+| Instrumentation | ~20-30 key functions | `#[instrument(skip_all)]` |
+
+## What We Don't Change
+
+- No changes to DQN/PPO trainer internals (instrument at fold level in binaries)
+- No changes to hot-path tick processing (no `#[instrument]` on SIMD/RDTSC paths)
+- No new Rust dependencies (tracing-opentelemetry already in workspace)
+- No changes to Tempo, Loki, or Prometheus deployment configs
+- Existing infra cockpit layout stays as-is (just one row added)
+
+## Risks and Mitigations
+
+- **nvidia-smi subprocess overhead**: 5s poll interval, `nvidia-smi` takes ~50ms. 1% overhead on a 5s window. Acceptable.
+- **nvidia-smi unavailable**: CPU-only dev machines don't have it. Poller logs a warning once and exits the thread gracefully.
+- **Trace volume from training**: Training runs are ~30-60min with ~1000 folds. At one span per fold, volume is negligible (~1000 spans per job).
+- **Metric cardinality**: `model` × `fold` labels. Max ~10 models × ~50 folds = 500 series. Low cardinality.