foxhunt/docs/plans/2026-03-01-training-observability-design.md

Training Binary Observability + Grafana Dashboards Design

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Instrument training binaries with GPU metrics polling, per-epoch Prometheus metrics, and OTLP tracing so we can validate whether the GPU is maxed out during real DQN/PPO training runs. Deploy Grafana dashboards to visualize everything.

Architecture: Training binaries poll nvidia-smi every 5s for GPU metrics, emit per-fold training metrics via existing Prometheus helpers, and export OTLP traces to Tempo. Grafana consumes Prometheus (metrics), Loki (logs), and Tempo (traces) with cross-linking.

Tech Stack: Rust tracing + tracing-opentelemetry (already in workspace), Prometheus client (already used), nvidia-smi CLI, Grafana 8.x dashboards (JSON ConfigMaps).

---

Problem Statement

The training binaries (train_baseline_rl, train_baseline_supervised, hyperopt_baseline_rl, hyperopt_baseline_supervised) have a Prometheus metrics server running on port 9094 with 18 training metrics registered, but:

1. Zero GPU metrics emitted — nvidia-smi is called once at startup for batch sizing, never polled during training
2. train_baseline_rl.rs is a metrics black box — DQN/PPO trainers are delegated to but never call back into common::metrics::training_metrics. Only record_data_load() and set_active_workers() are called
3. No OTLP tracing — training binaries use plain tracing_subscriber::fmt(), not init_observability()
4. No OTEL_EXPORTER_OTLP_ENDPOINT in K8s training job template
5. No Grafana dashboards for training or traces (design exists, implementation doesn't)

Data Flow

Training Binary → nvidia-smi poll (5s) → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana
Training Binary → set_epoch_loss() etc. → Prometheus gauges → /metrics :9094 → Prometheus scrape → Grafana
Training Binary → init_observability() → OTLP gRPC :4317 → Tempo → Grafana (Traces)
Training Binary → stdout → containerd → Promtail → Loki → Grafana (linked via trace_id)

Component 1: GPU Metrics Polling

New file: crates/common/src/metrics/gpu_metrics.rs

Metrics Registered

Metric Name	Type	Description
foxhunt_gpu_utilization_percent	Gauge	SM utilization 0-100
foxhunt_gpu_memory_used_bytes	Gauge	VRAM used in bytes
foxhunt_gpu_memory_total_bytes	Gauge	VRAM total in bytes
foxhunt_gpu_temperature_celsius	Gauge	GPU temperature
foxhunt_gpu_power_draw_watts	Gauge	Power consumption

Implementation

• poll_gpu_metrics() — shells out to nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv,noheader,nounits, parses CSV, sets gauges. MiB values from nvidia-smi are converted to bytes. Silently logs warning and skips update on parse failure (nvidia-smi not available on CPU-only machines).
• start_gpu_poller(interval_secs: u64) — spawns a daemon thread named "gpu-metrics-poller" that calls poll_gpu_metrics() in a loop with std::thread::sleep(Duration::from_secs(interval_secs)). Returns immediately. Thread is detached (no join handle needed).

Integration

All 4 training binaries call gpu_metrics::start_gpu_poller(5) in main() right after metrics_server::start_metrics_server(9094).

Component 2: Wire Per-Epoch Metrics into train_baseline_rl.rs

Since DQN/PPO trainers return only a single f64 (best validation loss), we instrument at the fold level in the binary, not inside the trainers.

Per-Fold Metrics Calls

After each DQN/PPO fold completes:

metrics::set_epoch(model_name, fold_idx, fold_idx as f64);  // fold as "epoch"
metrics::set_epoch_loss(model_name, fold_idx, best_val_loss);
metrics::set_validation_loss(model_name, fold_idx, best_val_loss);
metrics::set_iteration_seconds(model_name, fold_idx, fold_elapsed.as_secs_f64());

After data loading:

metrics::record_data_load(model_name, load_elapsed.as_secs_f64());  // already done

On checkpoint save (in the checkpoint callback):

metrics::record_checkpoint_save(model_name, fold_idx, ckpt_bytes.len(), save_elapsed);

Hyperopt Binaries

hyperopt_baseline_rl.rs and hyperopt_baseline_supervised.rs — add set_epoch() and set_epoch_loss() per trial iteration. Lower priority since hyperopt is exploratory, but the metrics are cheap.

Component 3: OTLP Tracing in Training Binaries

Change

In all 4 training binary main() functions, replace:

tracing_subscriber::fmt()
    .with_env_filter(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
    .init();

With:

common::observability::init_observability(
    "train_baseline_rl",  // service name varies per binary
    std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok().as_deref(),
);

When OTEL_EXPORTER_OTLP_ENDPOINT is unset, init_observability falls back to console-only logging (same as today). When set, spans flow to Tempo.

K8s Training Job Template

Add to infra/k8s/training/job-template.yaml pod env:

- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://tempo.foxhunt.svc.cluster.local:4317"

Component 4: Grafana Training Cockpit Dashboard

New file: infra/k8s/monitoring/dashboards/foxhunt-training-cockpit.json

Provisioned as a ConfigMap, auto-loaded by Grafana's dashboard provisioner (same pattern as existing foxhunt-cockpit.json).

Dashboard Layout (8 rows)

Per the existing design doc's Component 4 spec, using the actual metric names from common/src/metrics/training_metrics.rs and the new GPU metrics:

Row	Focus	Panels
1	Active Jobs	active_workers gauge, fold progress
2	Training Progress	epoch_loss + validation_loss timeseries, batches_per_second, iteration_seconds
3	GPU Resources	gpu_utilization_percent, gpu_memory_used/total, gpu_temperature, gpu_power_draw
4	Model Quality	eval_accuracy, eval_f1, eval_precision, eval_recall
5	Checkpoints	checkpoint_saves_total, checkpoint_failures_total, checkpoint_duration_seconds, checkpoint_size_bytes
6	Data Pipeline	batches_processed, data_load_seconds
7	Errors	nan_detected_total, gradient_explosion_total, feature_errors_total
8	Timing	iteration_seconds histogram, data_load_seconds histogram

Template Variables

• $model — dropdown: dqn, ppo, tft, etc. Filters all panels by model label
• $fold — dropdown: 0, 1, 2, etc. Filters by fold label
• $interval — auto interval for rate() queries

Component 5: Grafana Traces Dashboard

New file: infra/k8s/monitoring/dashboards/foxhunt-traces.json

• Service map panel (Tempo node graph)
• Trace search table (service, operation, duration, status filters)
• Trace duration histogram
• Error rate by service timeseries

Component 6: Existing Dashboard Updates

• foxhunt-cockpit.json — add one "Trace Health" row: spans/sec and error rate from Tempo metrics generator
• Service annotations — add Prometheus scrape annotations to 8 service deployment YAMLs (per existing design doc Component 1)

Component 7: Service OTLP Wiring

Per existing design doc Component 2: Add init_tracing() call to 8 service main() functions. Add #[instrument(skip_all)] to ~20-30 key async functions (gRPC handlers, DB queries, ML inference paths). No instrumentation on hot-path tick processing.

File Changes Summary

Category	Files	Change
New GPU metrics module	common/src/metrics/gpu_metrics.rs	nvidia-smi polling + 5 gauges
Training binary metrics	ml/examples/train_baseline_rl.rs	Wire per-fold metrics calls
Training binary OTLP	4 training binary main() functions	Replace fmt with init_observability
K8s training job	infra/k8s/training/job-template.yaml	Add OTEL endpoint env var
New dashboards	2 JSON files	training-cockpit, traces
Updated dashboards	1 JSON file	cockpit (trace row)
K8s service annotations	8 service YAMLs	Prometheus scrape annotations
Service OTLP	8 service main.rs	Add init_tracing()
Instrumentation	~20-30 key functions	#[instrument(skip_all)]

What We Don't Change

• No changes to DQN/PPO trainer internals (instrument at fold level in binaries)
• No changes to hot-path tick processing (no #[instrument] on SIMD/RDTSC paths)
• No new Rust dependencies (tracing-opentelemetry already in workspace)
• No changes to Tempo, Loki, or Prometheus deployment configs
• Existing infra cockpit layout stays as-is (just one row added)

Risks and Mitigations

• nvidia-smi subprocess overhead: 5s poll interval, nvidia-smi takes ~50ms. 1% overhead on a 5s window. Acceptable.
• nvidia-smi unavailable: CPU-only dev machines don't have it. Poller logs a warning once and exits the thread gracefully.
• Trace volume from training: Training runs are ~30-60min with ~1000 folds. At one span per fold, volume is negligible (~1000 spans per job).
• Metric cardinality: model × fold labels. Max ~10 models × ~50 folds = 500 series. Low cardinality.