foxhunt/docs/plans/2026-03-02-additional-training-metrics-design.md

Additional Training Metrics Design

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Add 17 new Prometheus metrics (12 always-on gauges + 5 opt-in verbose) to provide full training diagnostics and production monitoring visibility.

Architecture: Two-tier metrics system. Tier 1 metrics are zero-cost atomic gauge writes recorded in every trainer's epoch loop. Tier 2 metrics (histograms and per-layer stats) are gated behind FOXHUNT_VERBOSE_METRICS=1 and disabled by default. All metrics flow through the existing Prometheus → monitoring_service → gRPC → fxt CLI pipeline.

---

Current State

• 24 Prometheus metrics defined in crates/common/src/metrics/training_metrics.rs
• ~20 actively recorded across 5 training binaries
• Monitoring service queries all foxhunt_training_* and foxhunt_hyperopt_* via wildcard
• Proto has TrainingSession (19 fields) and GpuSnapshot (5 fields)

Tier 1: Always-On Gauges (12 new)

RL Diagnostics (6)

Metric Name	Labels	Source	Purpose
foxhunt_training_q_value_mean	model, fold	DQN epoch loop	Detect value overestimation
foxhunt_training_q_value_max	model, fold	DQN epoch loop	Detect divergence (exploding Q-values)
foxhunt_training_policy_entropy	model, fold	PPO epoch loop	Detect premature convergence (entropy → 0)
foxhunt_training_kl_divergence	model, fold	PPO epoch loop	Detect unstable policy updates
foxhunt_training_advantage_mean	model, fold	PPO epoch loop	Detect advantage bias (should be ~0)
foxhunt_training_replay_buffer_size	model, fold	DQN epoch loop	Monitor replay buffer occupancy

Gradient & Training Health (3)

Metric Name	Labels	Source	Purpose
foxhunt_training_gradient_norm	model, fold	All trainers post-backward	Trend before gradient explosion
foxhunt_training_learning_rate	model, fold	All trainers per epoch	Correlate LR schedule with loss plateaus
foxhunt_training_epoch_duration_seconds	model, fold	All trainers	Detect performance regressions

Hyperopt Intra-Trial Progress (3)

Metric Name	Labels	Source	Purpose
foxhunt_hyperopt_trial_epoch	model	Hyperopt adapters	Current epoch within active trial
foxhunt_hyperopt_trial_best_loss	model	Hyperopt adapters	Best loss in current trial
foxhunt_hyperopt_elapsed_seconds	model	Hyperopt binaries	Wall-clock time since start

Tier 2: Opt-In Verbose Metrics (5 new, FOXHUNT_VERBOSE_METRICS=1)

Metric Name	Type	Labels	Purpose
foxhunt_training_batch_loss	Histogram	model, fold	Per-batch loss distribution
foxhunt_training_gradient_norm_per_layer	Histogram	model, fold, layer	Per-layer gradient health
foxhunt_training_replay_priority_mean	Gauge	model, fold	PER staleness detection
foxhunt_training_advantage_std	Gauge	model, fold	Advantage normalization quality
foxhunt_training_value_explained_variance	Gauge	model, fold	Value function prediction quality

Gating Mechanism

// In training_metrics.rs
static VERBOSE: OnceLock<bool> = OnceLock::new();

pub fn verbose_enabled() -> bool {
    *VERBOSE.get_or_init(|| std::env::var("FOXHUNT_VERBOSE_METRICS").is_ok())
}

Recording calls for Tier 2 metrics check verbose_enabled() and return early if disabled.

Proto Changes (monitoring.proto)

Add to TrainingSession:

// RL diagnostics
float q_value_mean = 20;
float q_value_max = 21;
float policy_entropy = 22;
float kl_divergence = 23;
float advantage_mean = 24;
uint32 replay_buffer_size = 25;

// Gradient & training health
float gradient_norm = 26;
float learning_rate = 27;
float epoch_duration_seconds = 28;

// Hyperopt intra-trial
uint32 hyperopt_trial_epoch = 29;
float hyperopt_trial_best_loss = 30;
float hyperopt_elapsed_seconds = 31;

Service Changes

monitoring_service/src/service.rs

Extend group_into_sessions match arms for all 12 new metric names → proto fields. No new Prometheus queries needed (wildcard already covers foxhunt_training_* and foxhunt_hyperopt_*).

monitoring_service/src/prometheus_client.rs

No changes needed.

CLI Changes (fxt train monitor)

Main table: add gradient_norm column

Model   Fold  Epoch  Loss     Val Loss  Batch/s  Grad Norm  Eval Acc
----------------------------------------------------------------------
dqn     0     25     0.0234   0.0312    148.2    1.23       -
ppo     0     12     0.0156   0.0089    92.1     0.87       -

RL diagnostics section (only when non-zero)

RL Diagnostics:
  dqn: Q-mean=12.3 Q-max=45.6 | buffer=50000/100000
  ppo: entropy=1.23 KL=0.008 adv-mean=0.001

Hyperopt intra-trial (inline enhancement)

Hyperopt (dqn): trial 3/20 (epoch 12, loss 0.042) | best Sharpe 1.23 | 0 failures

Existing Gap Fills

Wire up eval metrics (accuracy, precision, recall, F1) in DQN and PPO trainers. These proto fields already exist but are never populated for RL models.

Testing

• Extend training_metrics.rs unit tests for new registrations + verbose gating
• Extend service.rs group_into_sessions tests with new metric → proto field mappings
• Full pipeline tested on next CI build with new training binaries

Files Modified

File	Change
crates/common/src/metrics/training_metrics.rs	+12 gauges, +5 verbose metrics, verbose gate
services/monitoring_service/proto/monitoring.proto	+12 proto fields
services/monitoring_service/src/service.rs	+12 match arms in group_into_sessions
bin/fxt/src/commands/train/monitor.rs	Enhanced display (grad norm col, RL section, hyperopt detail)
crates/ml/src/trainers/dqn/trainer.rs	Record Q-values, gradient norm, LR, epoch duration, eval metrics, buffer size
crates/ml/src/trainers/ppo.rs	Record entropy, KL, advantage, gradient norm, LR, epoch duration, eval metrics
crates/ml/examples/train_baseline_supervised.rs	Record gradient norm, LR, epoch duration
crates/ml/examples/hyperopt_baseline_rl.rs	Record trial_epoch, trial_best_loss, elapsed_seconds
crates/ml/examples/hyperopt_baseline_supervised.rs	Record trial_epoch, trial_best_loss, elapsed_seconds